You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Xianxun Ye <ye...@gmail.com> on 2023/04/02 14:58:12 UTC
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Hi Feng,
Thanks for driving this Flip, I do believe this Flip could be helpful for users. The firm I work for also manages a lot of catalogs, and the submission of tasks becomes slow because of loading a number of catalogs. We obtain the catalogs in the user's Flink SQL through regular expression to avoid loading all catalogs to improve the speed of task submission. After reading flip-295, I have some questions:
1. When creating a catalog with CREATE CATALOG, the asynchronous registration method is used by default. What if asynchronous registration failed? And how to implement asynchronous registration?
2. I also have the same question with Shammon’s first comment. The Map<String, Catalog> catalogs and CatalogStore catalogstore in the CatalogManager all have the same catalog name, but correponding to different catalog instances. For example: catalog1-> jdbcCatalog in the catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would this case happen? Maybe we should define `catalogs` and `catalogstore` more clearly.
Best regards,
Xianxun
> 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
>
> MemoryCatalogStore
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi everyone,
Thank you everyone for your valuable input. If there are no further
questions or concerns about the FLIP-295[1], I would like to start voting
tomorrow (6/14).
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
Best,
Feng
On Mon, Jun 12, 2023 at 10:54 AM Feng Jin <ji...@gmail.com> wrote:
> hi, Hang
>
> Thanks for your reply, Make sense to me, I updated the FLIP.
>
>
> Best,
>
> Feng
>
>
>
> On Fri, Jun 9, 2023 at 3:34 PM Hang Ruan <ru...@gmail.com> wrote:
>
>> Hi, Feng.
>>
>> I reviewed this FLIP again and found some little places that may need to
>> optimize.
>>
>> 1. `CatalogDescriptor` may need a private constructor.
>> 2. The method `initCatalog` in `CatalogManager` is not introduced.
>>
>> Best,
>> Hang
>>
>> Feng Jin <ji...@gmail.com> 于2023年6月6日周二 21:17写道：
>>
>> > Hi Leonard,
>> >
>> > Thanks for your reply.
>> >
>> > > 1. a How to construct a CatalogDescriptor ?
>> >
>> > I think it would be helpful to add a method for constructing a
>> > CatalogDescriptor, as you mentioned in 1.c. I will update the
>> documentation
>> > later.
>> >
>> > > 1.b How to visit the fields ? Could we use Configuration instead of
>> > Map<String, String> ?
>> >
>> > I believe that the use of Map<String, String> options is only intended
>> for
>> > creating a catalog and not for accessing internal parameters.
>> >
>> > Since all of the relevant parameters for CREATE CATALOG are also stored
>> in
>> > Map<String, String> options, my understanding is that using Map<String,
>> > String> options should suffice.
>> >
>> > Here is the implementation of execute CREATE CATALOG statement.
>> > ```java
>> > private TableResultInternal createCatalog(CreateCatalogOperation
>> operation)
>> > {
>> > String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
>> > try {
>> > String catalogName = operation.getCatalogName();
>> > Map<String, String> properties = operation.getProperties();
>> >
>> > Catalog catalog =
>> > FactoryUtil.createCatalog(
>> > catalogName,
>> > properties,
>> > tableConfig,
>> > resourceManager.getUserClassLoader());
>> > catalogManager.registerCatalog(catalogName, catalog);
>> >
>> > return TableResultImpl.TABLE_RESULT_OK;
>> > } catch (CatalogException e) {
>> > throw new ValidationException(exMsg, e);
>> > }
>> > }
>> > ```
>> >
>> >
>> > > 2. Do we have plan to offer a default CatalogStore if user didn’t
>> config
>> > this?
>> >
>> > Yes, the in-memory catalogStore will be used as the default CatalogStore
>> > even if the user has not configured one
>> >
>> >
>> > Best,
>> > Feng
>> >
>> >
>> > On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xb...@gmail.com> wrote:
>> >
>> > > Hi, Feng
>> > >
>> > > Sorry for reply late, but I’ve some comments about the FLIP
>> > >
>> > >
>> > > 1. The introduced Public class CatalogDescriptor seems missed some
>> > > necessary component
>> > > a) How to construct a CatalogDescriptor ?
>> > > b) How to visit the fields ? Could we use Configuration instead of
>> > > Map<String, String> ?
>> > > c) Could we offer a built-in factory method to build a
>> > CatalogDescriptor
>> > > like
>> > > public static CatalogDescriptor of(String catalogName,
>> Configuration
>> > > configuration)
>> > >
>> > > 2. The FLIP said “By default, there are two built-in CatalogStores
>> > > available: the In-Memory CatalogStore and the File CatalogStore” ,
>> > > Do we have plan to offer a default CatalogStore if user didn’t config
>> > > this? IIUC, users can obtains the benefits from lazy catalog
>> > > initialization If
>> > > we have a default catalogstore even it is in-memory.
>> > >
>> > > Best,
>> > > Leonard
>> > >
>> > >
>> > >
>> > > > On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
>> > > >
>> > > > Hi everyone,
>> > > >
>> > > > Thank you everyone for your valuable input. If there are no further
>> > > > questions or concerns about the FLIP[1], I would like to start
>> voting
>> > > > tomorrow (6/7).
>> > > >
>> > > >
>> > > > [1]
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > >
>> > > >
>> > > > Best,
>> > > > Feng
>> > > >
>> > > >
>> > > > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com>
>> wrote:
>> > > >
>> > > >> Hi Samrat,
>> > > >>
>> > > >> Thanks for your advice.
>> > > >>
>> > > >>> 1. The createCatalog method does not mention any exceptions being
>> > > >> thrown.
>> > > >>
>> > > >> CreateCatalog will throw CatalogException like registerCatalog. As
>> > > >> CatalogException is a RuntimeException,
>> > > >> there is no explicit declaration of throwing Exceptions in
>> > > CatalogManager
>> > > >> and TableEnvironment.
>> > > >> To avoid misunderstandings, I have added the "throw
>> CatalogException"
>> > > flag
>> > > >> to the createCatalog method definition of CatalogStore.
>> > > >>
>> > > >>> 2. Could you please provide an exhaustive list of the supported
>> > kinds?
>> > > >>
>> > > >> Sure, the documentation now includes the configuration of the
>> > built-in
>> > > >> CatalogStore as well as how to configure a custom CatalogStore.
>> > > >>
>> > > >>
>> > > >> Best,
>> > > >> Feng
>> > > >>
>> > > >>
>> > > >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com>
>> > > wrote:
>> > > >>
>> > > >>> Hi Feng,
>> > > >>>
>> > > >>> Thank you for providing the proposal. I believe this feature will
>> be
>> > > >>> highly
>> > > >>> valuable.
>> > > >>>
>> > > >>> I have a couple of inquiries:
>> > > >>>
>> > > >>> 1. According to the documentation [1], the createCatalog method
>> does
>> > > not
>> > > >>> mention any exceptions being thrown. However, I would like to
>> confirm
>> > > if
>> > > >>> it
>> > > >>> is always true that there will be no failures in all scenarios.
>> > Please
>> > > let
>> > > >>> me know if there is any additional information I may have missed.
>> > > >>>
>> > > >>> 2. Regarding the registration process using the
>> > > `table.catalog-store.kind`
>> > > >>> configuration, could you please provide an exhaustive list of the
>> > > >>> supported
>> > > >>> kinds?
>> > > >>> It would be great to have a comprehensive understanding of the
>> > > options
>> > > >>> available.
>> > > >>>
>> > > >>> Bests,
>> > > >>> Samrat
>> > > >>>
>> > > >>> [1]
>> > > >>>
>> > > >>>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > >>>
>> > > >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com>
>> > wrote:
>> > > >>>
>> > > >>>> Hi Hang and Jark
>> > > >>>>
>> > > >>>> Thank you very much for your reply.
>> > > >>>>
>> > > >>>> @Jark
>> > > >>>>> 1. Could you move the CatalogStore registration API to the
>> "Public
>> > > >>>> Interface" section?
>> > > >>>>
>> > > >>>> Indeed, it is more reasonable.
>> > > >>>>
>> > > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
>> > > >>>>
>> > > >>>> Sure, the names that you provided are indeed more logical and
>> > easier
>> > > >>> to
>> > > >>>> understand.
>> > > >>>>
>> > > >>>>> 3. About the open/close method in CatalogStoreFactory.
>> > > >>>>
>> > > >>>> I also agree with the proposed requirement scenario and design. I
>> > have
>> > > >>>> already made modifications to the interface.
>> > > >>>>
>> > > >>>> @Hang
>> > > >>>>
>> > > >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
>> > > close
>> > > >>>> the resource.
>> > > >>>>
>> > > >>>> I have already added the missing method.
>> > > >>>>
>> > > >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>> > > >>>>
>> > > >>>> Sorry for the misspelling, I updated this part.
>> > > >>>>
>> > > >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
>> > > methods
>> > > >>>> to init or close its resource
>> > > >>>>
>> > > >>>> This is indeed a scenario that needs to be considered, and I have
>> > > added
>> > > >>>> relevant methods accordingly.
>> > > >>>> Additionally, I have included some relevant description
>> information
>> > in
>> > > >>> the
>> > > >>>> documentation to help others understand
>> > > >>>> why the open/close methods need to be added.
>> > > >>>>
>> > > >>>>
>> > > >>>> Best,
>> > > >>>> Feng
>> > > >>>>
>> > > >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com>
>> wrote:
>> > > >>>>
>> > > >>>>> Hi Jing,
>> > > >>>>>
>> > > >>>>> Thank you for the update.
>> > > >>>>>
>> > > >>>>> 1. Could you move the CatalogStore registration API to the
>> "Public
>> > > >>>>> Interface" section?
>> > > >>>>> "Proposed Changes" is more like a place to describe the
>> > > implementation
>> > > >>>>> details.
>> > > >>>>>
>> > > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
>> > > >>> Besides,
>> > > >>>>> the config key
>> > > >>>>> name should be hierarchical[1]. Therefore, it may be better to
>> use:
>> > > >>>>> "table.catalog-store.kind"
>> > > >>>>> "table.catalog-store.file.path"
>> > > >>>>>
>> > > >>>>> 3. I think Hang's suggestions make sense.
>> > > >>>>>
>> > > >>>>> Others look good to me.
>> > > >>>>>
>> > > >>>>> Best,
>> > > >>>>> Jark
>> > > >>>>>
>> > > >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
>> > > >>> wrote:
>> > > >>>>>
>> > > >>>>>> Hi, Feng.
>> > > >>>>>>
>> > > >>>>>> Thanks for the update.
>> > > >>>>>> The current design looks good to me. I have some minor
>> comments.
>> > > >>>>>>
>> > > >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init
>> or
>> > > >>> close
>> > > >>>>> the
>> > > >>>>>> resource. For example, when storing the information in MySQL,
>> the
>> > > >>> store
>> > > >>>>>> needs to open and close the connections.
>> > > >>>>>>
>> > > >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>> > > >>>>>>
>> > > >>>>>> 3. About the usage in the sql gateway.
>> > > >>>>>> Considering the usage in sql gateway, the sql gateway may
>> create a
>> > > >>>>>> CatalogStore for each session.
>> > > >>>>>> If we are using the MySqlCatalogStore, there would be so many
>> > > >>>>> connections.
>> > > >>>>>> How can we reuse the connection among these sessions?
>> > > >>>>>> I think sql gateway need to maintain a connection pool in
>> > > >>>>>> the CatalogStoreFactory and each session get its own connection
>> > from
>> > > >>>> the
>> > > >>>>>> pool when it is created.
>> > > >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close`
>> methods
>> > > >>> to
>> > > >>>>> init
>> > > >>>>>> or close its resource.
>> > > >>>>>> Or is there a better way?
>> > > >>>>>>
>> > > >>>>>> Best,
>> > > >>>>>> Hang
>> > > >>>>>>
>> > > >>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
>> > > >>>>>>
>> > > >>>>>>> Thanks Jingsong.
>> > > >>>>>>>
>> > > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
>> > > >>>>>>>
>> > > >>>>>>> +1 For this, I have already updated FLIP.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> Best,
>> > > >>>>>>> Feng
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
>> > > >>> jingsonglee0@gmail.com>
>> > > >>>>>>> wrote:
>> > > >>>>>>>
>> > > >>>>>>>> Thanks Feng,
>> > > >>>>>>>>
>> > > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see
>> many
>> > > >>>>>>>> functions are converted to createxxx from registerxxx.
>> > > >>>>>>>>
>> > > >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <
>> jinfeng1995@gmail.com
>> > > >>>>
>> > > >>>>>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>> Hi jark, thanks for your suggestion.
>> > > >>>>>>>>>
>> > > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
>> > > >>> the
>> > > >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
>> > > >>>>> Otherwise,
>> > > >>>>>>>> there
>> > > >>>>>>>>> might be some data inconsistencies when CatalogStore is
>> > > >>> changed.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
>> > > >>>>>> TableEnv.
>> > > >>>>>>>>> Therefore, my current proposal is to add a method to
>> configure
>> > > >>>> the
>> > > >>>>>>>>> CatalogStore in EnvironmentSettings.
>> > > >>>>>>>>>
>> > > >>>>>>>>> // Initialize a catalog Store
>> > > >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
>> > > >>>>>>>>>
>> > > >>>>>>>>> // set up the Table API
>> > > >>>>>>>>> final EnvironmentSettings settings =
>> > > >>>>>>>>>
>> > > >>> EnvironmentSettings.newInstance().inBatchMode()
>> > > >>>>>>>>> .withCatalogStore(catalogStore)
>> > > >>>>>>>>> .build();
>> > > >>>>>>>>>
>> > > >>>>>>>>> final TableEnvironment tableEnv =
>> > > >>>>>>>> TableEnvironment.create(settings);
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
>> > > >>>> default
>> > > >>>>>>>> method,
>> > > >>>>>>>>> not an interface method?
>> > > >>>>>>>>>
>> > > >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
>> > > >>> that
>> > > >>>> as
>> > > >>>>> a
>> > > >>>>>>> new
>> > > >>>>>>>>> interface, the CatalogStoreFactory should not have a default
>> > > >>>> method
>> > > >>>>>> but
>> > > >>>>>>>> an
>> > > >>>>>>>>> interface method. I have already modified the interface.
>> > > >>>>>>>>>
>> > > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
>> > > >>>>>>>>> deprecated`registerCatalog`.
>> > > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
>> > > >>> mention
>> > > >>>> the
>> > > >>>>>>>> changed
>> > > >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
>> > > >>>> other
>> > > >>>>>>>>> persisted catalog store) is used.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> [1].
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> Best,
>> > > >>>>>>>>> Feng
>> > > >>>>>>>>>
>> > > >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
>> > > >>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
>> > > >>>>>>>>>> The current design looks pretty good to me. I just have
>> some
>> > > >>>>> minor
>> > > >>>>>>>>>> comments.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
>> > > >>> the
>> > > >>>>>>>> CatalogStore
>> > > >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
>> > > >>> there
>> > > >>>>>> might
>> > > >>>>>>> be
>> > > >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
>> > > >>>> default
>> > > >>>>>>>> method,
>> > > >>>>>>>>>> not an interface method?
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
>> > > >>>>> deprecated
>> > > >>>>>>>>>> `registerCatalog`.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
>> > > >>> mention
>> > > >>>> the
>> > > >>>>>>>> changed
>> > > >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore
>> (or
>> > > >>>>> other
>> > > >>>>>>>>>> persisted catalog store) is used.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Best,
>> > > >>>>>>>>>> Jark
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
>> > > >>> jinfeng1995@gmail.com>
>> > > >>>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> @Ron
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
>> > > >>>> method,
>> > > >>>>>> do
>> > > >>>>>>> we
>> > > >>>>>>>>>> need
>> > > >>>>>>>>>>> to provide a default implementation?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Yes, we will provide a default
>> > > >>> InMemoryCatalogStoreFactory to
>> > > >>>>>>> create
>> > > >>>>>>>> an
>> > > >>>>>>>>>>> InMemoryCatalogStore.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
>> > > >>> initializing
>> > > >>>>> it,
>> > > >>>>>>>> whether
>> > > >>>>>>>>>> we
>> > > >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Yes, in the current design, catalogs are stored as
>> > > >>> snapshots,
>> > > >>>>> and
>> > > >>>>>>>> once
>> > > >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
>> > > >>>>>> Catalog>
>> > > >>>>>>>>>>> catalogs.
>> > > >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
>> > > >>> the
>> > > >>>>>> primary
>> > > >>>>>>>>>> source
>> > > >>>>>>>>>>> for obtaining the corresponding Catalog.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
>> > > >>>>>>>>>>> `catalog.store.path`?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
>> > > >>> seem a
>> > > >>>>> bit
>> > > >>>>>>>>>> strange. I
>> > > >>>>>>>>>>> will update the FLIP.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> @Shammon
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Thank you for the review. I have made the necessary
>> > > >>>>> corrections.
>> > > >>>>>>>>>>> Regarding the modifications made to the Public Interface,
>> > > >>> I
>> > > >>>>> have
>> > > >>>>>>> also
>> > > >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Best,
>> > > >>>>>>>>>>> Feng
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
>> > > >>>> zjureel@gmail.com>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> Hi feng,
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Thanks for updating, I have some minor comments
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
>> > > >>>>>> `Public
>> > > >>>>>>>>>>>> Interfaces`, it is not a public interface.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
>> > > >>> and
>> > > >>>>>>>>>>>> `CatalogStoreFactory`
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
>> > > >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
>> > > >>>>> should
>> > > >>>>>> be
>> > > >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
>> > > >>>>>>>>>>>> catalogStore.get(catalogName);`
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>> Shammon FY
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
>> > > >>>> ron9.liu@gmail.com>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Hi, Feng
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
>> > > >>>> useful
>> > > >>>>>> for
>> > > >>>>>>>>>> catalog
>> > > >>>>>>>>>>>>> management.
>> > > >>>>>>>>>>>>> I have some small questions:
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> 1. Regarding the
>> > > >>> CatalogStoreFactory#createCatalogStore
>> > > >>>>>> method,
>> > > >>>>>>>> do we
>> > > >>>>>>>>>>>> need
>> > > >>>>>>>>>>>>> to provide a default implementation?
>> > > >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
>> > > >>>> initializing
>> > > >>>>>> it,
>> > > >>>>>>>>>> whether
>> > > >>>>>>>>>>> we
>> > > >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
>> > > >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
>> > > >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
>> > > >>> them to
>> > > >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>> Ron
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
>> > > >>> 21:19写道：
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Hi yuxia
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
>> > > >>>>> register
>> > > >>>>>>> the
>> > > >>>>>>>>>>> Catalog,
>> > > >>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>> initialize it and open it. right?
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
>> > > >>>>>> original
>> > > >>>>>>>>>> CREATE
>> > > >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
>> > > >>>>>>>>>> registerCatalog
>> > > >>>>>>>>>>> so
>> > > >>>>>>>>>>>>>> that parameter validation can be performed.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
>> > > >>>>>>> reflected
>> > > >>>>>>>> in
>> > > >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
>> > > >>>>> catalog
>> > > >>>>>>>>>>>>> configurations,
>> > > >>>>>>>>>>>>>> only initialization is required in getCatalog.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
>> > > >>>>>>>> luoyuxia@alumni.sjtu.edu.cn
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Hi, Feng.
>> > > >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
>> > > >>>>>>>> initialization*.
>> > > >>>>>>>>>>> If
>> > > >>>>>>>>>>>>> i'm
>> > > >>>>>>>>>>>>>>> wrong, please correct me.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
>> > > >>>> access
>> > > >>>>>> the
>> > > >>>>>>>>>>> catalog,
>> > > >>>>>>>>>>>>> then
>> > > >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
>> > > >>>>> Changes,
>> > > >>>>>>>> once we
>> > > >>>>>>>>>>>>>> register
>> > > >>>>>>>>>>>>>>> the Catalog,
>> > > >>>>>>>>>>>>>>> we initialize it and open it. right?
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>> Yuxia
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> ----- 原始邮件 -----
>> > > >>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
>> > > >>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
>> > > >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
>> > > >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
>> > > >>>>>> Catalog
>> > > >>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>> and asynchronous registration
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> One of the major changes is that current design
>> > > >>> will
>> > > >>>>>>> provide
>> > > >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
>> > > >>>> of a
>> > > >>>>>>>> cache,
>> > > >>>>>>>>>>> which
>> > > >>>>>>>>>>>>>> means
>> > > >>>>>>>>>>>>>>> once it has been initialized, any changes done by
>> > > >>>> other
>> > > >>>>>>>> sessions
>> > > >>>>>>>>>>> will
>> > > >>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
>> > > >>>>>> further
>> > > >>>>>>>>>>>> improvement.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
>> > > >>>>>>>> jinfeng1995@gmail.com>
>> > > >>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
>> > > >>>>>> progress
>> > > >>>>>>>> of
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
>> > > >>> Shammon
>> > > >>>> FY,
>> > > >>>>>>>> ShengKai
>> > > >>>>>>>>>>> Fang
>> > > >>>>>>>>>>>>>> and I
>> > > >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
>> > > >>>>>> solution
>> > > >>>>>>>> for
>> > > >>>>>>>>>>> Flink
>> > > >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
>> > > >>>> have
>> > > >>>>>>>> updated
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>> final
>> > > >>>>>>>>>>>>>>>> solution in FLIP.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> 1.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
>> > > >>>>>>>> configuration
>> > > >>>>>>>>>>>> similar
>> > > >>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>> TableDescriptor.
>> > > >>>>>>>>>>>>>>>> 2.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
>> > > >>>>>>>>>>> storeCatalog(String
>> > > >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
>> > > >>>>>> CatalogDescriptor
>> > > >>>>>>>>>>>>>>> getCatalog(String)
>> > > >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
>> > > >>>> Catalog
>> > > >>>>>>>> instance.
>> > > >>>>>>>>>>>> This
>> > > >>>>>>>>>>>>>> way,
>> > > >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
>> > > >>> saving
>> > > >>>>> and
>> > > >>>>>>>>>>> retrieving
>> > > >>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>> configurations without having to initialize
>> > > >>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>> 3.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> The default registerCatalog(String
>> > > >>> catalogName,
>> > > >>>>>>> Catalog
>> > > >>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
>> > > >>>>>>> deprecated.
>> > > >>>>>>>>>>>>>>>> 4.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> A new function registerCatalog(String
>> > > >>>> catalogName,
>> > > >>>>>>>>>>>>> CatalogDescriptor
>> > > >>>>>>>>>>>>>>>> catalog) will be added to serve as the
>> > > >>> default
>> > > >>>>>>>> registration
>> > > >>>>>>>>>>>>> function
>> > > >>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>> catalogs in CatalogManager.
>> > > >>>>>>>>>>>>>>>> 5.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
>> > > >>>> will
>> > > >>>>>>>> remain
>> > > >>>>>>>>>>>>> unchanged
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>> save initialized catalogs.This means that
>> > > >>>> deletion
>> > > >>>>>>>>>> operations
>> > > >>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>> one
>> > > >>>>>>>>>>>>>>>> session won't synchronize with other
>> > > >>> sessions.
>> > > >>>>>>>>>>>>>>>> 6.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> To support multi-session synchronization
>> > > >>>> scenarios
>> > > >>>>>> for
>> > > >>>>>>>>>>> deletions
>> > > >>>>>>>>>>>>>> later
>> > > >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
>> > > >>>>>>>>>>> configurable.There
>> > > >>>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>> three possible situations:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
>> > > >>>>>>>> automatically
>> > > >>>>>>>>>>> clear
>> > > >>>>>>>>>>>>>>>> long-unused catalogs.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
>> > > >>>>>> getCatalog
>> > > >>>>>>>>>>> creates a
>> > > >>>>>>>>>>>>> new
>> > > >>>>>>>>>>>>>>>> instance.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> This is the document for discussion:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> This is the final proposal document:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Thank you very much for your attention and
>> > > >>>>> suggestions
>> > > >>>>>> on
>> > > >>>>>>>> this
>> > > >>>>>>>>>>>>> FLIP. A
>> > > >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
>> > > >>> on
>> > > >>>>> the
>> > > >>>>>>>> entire
>> > > >>>>>>>>>>>> design
>> > > >>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>> organizing offline discussions.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
>> > > >>>> about
>> > > >>>>>>> this
>> > > >>>>>>>> FLIP
>> > > >>>>>>>>>>>>> please
>> > > >>>>>>>>>>>>>>> feel
>> > > >>>>>>>>>>>>>>>> free to share.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
>> > > >>>>>>>>>>> <jing@ververica.com.invalid
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
>> > > >>> to
>> > > >>>> me.
>> > > >>>>>> We
>> > > >>>>>>>> could
>> > > >>>>>>>>>>>> still
>> > > >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
>> > > >>>> common
>> > > >>>>>>>> built-in
>> > > >>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
>> > > >>> every
>> > > >>>>>>>>>> CatalogSotre
>> > > >>>>>>>>>>>>>>>>> implementation has to take care of it.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
>> > > >>>>>>>>>> jinfeng1995@gmail.com
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Hi Jing,
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Thanks for your reply.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
>> > > >>>>> suggest
>> > > >>>>>>> you
>> > > >>>>>>>>>>>>> completely
>> > > >>>>>>>>>>>>>>> walk
>> > > >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
>> > > >>> the
>> > > >>>>> same
>> > > >>>>>>>> time, I
>> > > >>>>>>>>>>>>> greatly
>> > > >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
>> > > >>>> thoroughly
>> > > >>>>>>>> checked
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>> entire
>> > > >>>>>>>>>>>>>>>> FLIP
>> > > >>>>>>>>>>>>>>>>>> and made corrections to these issues.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
>> > > >>> FLIP
>> > > >>>>>>> design,
>> > > >>>>>>>>>>>>>>> CatalogManager
>> > > >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
>> > > >>> being
>> > > >>>>>>>> configured.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
>> > > >>> was
>> > > >>>> not
>> > > >>>>>>>> necessary
>> > > >>>>>>>>>>>>> because
>> > > >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
>> > > >>>> catalogs
>> > > >>>>>> to
>> > > >>>>>>>> store
>> > > >>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>> instances.
>> > > >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
>> > > >>>>>>> Therefore, I
>> > > >>>>>>>>>>>> modified
>> > > >>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>> part
>> > > >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
>> > > >>> Catalog>
>> > > >>>>>>> catalogs
>> > > >>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>>>> CatalogManager.
>> > > >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
>> > > >>> serve
>> > > >>>> as
>> > > >>>>>> the
>> > > >>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
>> > > >>>>>>>> functionality of
>> > > >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
>> > > >>>>>>>>>>>>>>>>>> The previous plan that kept
>> > > >>>>>>>> Map<String,Catalog>catalogs has
>> > > >>>>>>>>>>>> been
>> > > >>>>>>>>>>>>>>> moved
>> > > >>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>> Rejected Alternatives.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
>> > > >>>>>>>>>>>>> <jing@ververica.com.invalid
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> There are still many places contain
>> > > >>>>> inconsistent
>> > > >>>>>>>> content,
>> > > >>>>>>>>>>>> e.g.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
>> > > >>> used.
>> > > >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
>> > > >>>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
>> > > >>> not
>> > > >>>>> tell
>> > > >>>>>>>> what the
>> > > >>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>> doing.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
>> > > >>>>> suggest
>> > > >>>>>>> you
>> > > >>>>>>>>>>>> completely
>> > > >>>>>>>>>>>>>>> walk
>> > > >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
>> > > >>> issues.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
>> > > >>> mean
>> > > >>>>> that
>> > > >>>>>>> you
>> > > >>>>>>>> will
>> > > >>>>>>>>>>>> build
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
>> > > >>> is a
>> > > >>>>> very
>> > > >>>>>>>>>> different
>> > > >>>>>>>>>>>>>> design
>> > > >>>>>>>>>>>>>>>>>> concept
>> > > >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
>> > > >>> I am
>> > > >>>>> not
>> > > >>>>>>>>>> mistaken,
>> > > >>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
>> > > >>>> work
>> > > >>>>>>>> without
>> > > >>>>>>>>>>>> Optional
>> > > >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
>> > > >>>>> reason
>> > > >>>>>>>> why I
>> > > >>>>>>>>>>>>> mentioned
>> > > >>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
>> > > >>>>> Optional
>> > > >>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>> correct.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
>> > > >>>>>>>>>>>>> jinfeng1995@gmail.com>
>> > > >>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Hi Jing
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
>> > > >>>> FLIP
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
>> > > >>>> review. I
>> > > >>>>>>> have
>> > > >>>>>>>>>>> already
>> > > >>>>>>>>>>>>> made
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> relevant changes.
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
>> > > >>> the
>> > > >>>>>>>>>> multi-instance
>> > > >>>>>>>>>>>>>>>> scenarios, I
>> > > >>>>>>>>>>>>>>>>>> am
>> > > >>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
>> > > >>> multiple
>> > > >>>>>>>> scenarios
>> > > >>>>>>>>>> with
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> current
>> > > >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
>> > > >>>>> remove
>> > > >>>>>>>>>>> 'Map<String,
>> > > >>>>>>>>>>>>>>>> Catalog>'
>> > > >>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
>> > > >>>>>>>>>> InMemoryCatalogStore.
>> > > >>>>>>>>>>>> This
>> > > >>>>>>>>>>>>>>> way,
>> > > >>>>>>>>>>>>>>>>>> users
>> > > >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
>> > > >>>>> achieve
>> > > >>>>>>>>>>>>> multi-instance
>> > > >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
>> > > >>>> think?
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
>> > > >>>>>>>>>>>>>>> <jing@ververica.com.invalid
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
>> > > >>> are
>> > > >>>>>> still
>> > > >>>>>>>> some
>> > > >>>>>>>>>>> NIT
>> > > >>>>>>>>>>>>>> issues
>> > > >>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> FLIP
>> > > >>>>>>>>>>>>>>>>>>>>> like:
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
>> > > >>>> has
>> > > >>>>>> been
>> > > >>>>>>>> used
>> > > >>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
>> > > >>> example.
>> > > >>>> It
>> > > >>>>>>>> should be
>> > > >>>>>>>>>>>> fine
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> use
>> > > >>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
>> > > >>> after
>> > > >>>> you
>> > > >>>>>>>> submit
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>> PR.
>> > > >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
>> > > >>>> somewhere
>> > > >>>>> in
>> > > >>>>>>> the
>> > > >>>>>>>>>>>> rejected
>> > > >>>>>>>>>>>>>>>> section
>> > > >>>>>>>>>>>>>>>>>>> which
>> > > >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
>> > > >>> it
>> > > >>>>>>>> consistent.
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
>> > > >>>>>>>> multi-instance
>> > > >>>>>>>>>>>>>>> scenarios,
>> > > >>>>>>>>>>>>>>>> I
>> > > >>>>>>>>>>>>>>>>> am
>> > > >>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
>> > > >>> behaviour.
>> > > >>>> If
>> > > >>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
>> > > >>>> once
>> > > >>>>>> the
>> > > >>>>>>>>>> related
>> > > >>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>> has
>> > > >>>>>>>>>>>>>>>>>>> been
>> > > >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
>> > > >>> another
>> > > >>>>>>>> instance.
>> > > >>>>>>>>>> Did
>> > > >>>>>>>>>>> I
>> > > >>>>>>>>>>>>> miss
>> > > >>>>>>>>>>>>>>>>>>> something?
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
>> > > >>> Jin
>> > > >>>> <
>> > > >>>>>>>>>>>>>>> jinfeng1995@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> @Jing
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
>> > > >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
>> > > >>> name, I
>> > > >>>>>> have
>> > > >>>>>>>>>> updated
>> > > >>>>>>>>>>> it
>> > > >>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>> documentation.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
>> > > >>> should
>> > > >>>> be
>> > > >>>>>>>>>> implemented
>> > > >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
>> > > >>>>>> alternative
>> > > >>>>>>> 1
>> > > >>>>>>>>>> turns
>> > > >>>>>>>>>>>> out
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
>> > > >>> this
>> > > >>>>> part
>> > > >>>>>>> of
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> content.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
>> > > >>>> forward,
>> > > >>>>>> the
>> > > >>>>>>>>>> option 2
>> > > >>>>>>>>>>>>>>> solution
>> > > >>>>>>>>>>>>>>>>>> looks
>> > > >>>>>>>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
>> > > >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
>> > > >>>>> offline,
>> > > >>>>>>> we
>> > > >>>>>>>>>> think
>> > > >>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>> solution
>> > > >>>>>>>>>>>>>>>>>> 2
>> > > >>>>>>>>>>>>>>>>>>>>> might
>> > > >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
>> > > >>>>>>>> inconsistency
>> > > >>>>>>>>>>>> issues.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
>> > > >>>>> descriptions
>> > > >>>>>> in
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> content.
>> > > >>>>>>>>>>>>>>>> Would
>> > > >>>>>>>>>>>>>>>>>> you
>> > > >>>>>>>>>>>>>>>>>>>>> like
>> > > >>>>>>>>>>>>>>>>>>>>>> to clean them up?
>> > > >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
>> > > >>>> document
>> > > >>>>>> and
>> > > >>>>>>>>>>>> appreciate
>> > > >>>>>>>>>>>>>> your
>> > > >>>>>>>>>>>>>>>>>>>>> suggestions.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> @Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
>> > > >>> in
>> > > >>>>>>>> `Rejected
>> > > >>>>>>>>>>>>>>> Alternatives`
>> > > >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
>> > > >>> `Rejected
>> > > >>>>>>>>>> Alternatives`.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> Best
>> > > >>>>>>>>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
>> > > >>> Shammon
>> > > >>>>> FY
>> > > >>>>>> <
>> > > >>>>>>>>>>>>>>> zjureel@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
>> > > >>>>>> `Proposed
>> > > >>>>>>>>>>> Changes`,
>> > > >>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>> you
>> > > >>>>>>>>>>>>>>>>> put
>> > > >>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
>> > > >>>>>>>> Alternatives`? I
>> > > >>>>>>>>>>> think
>> > > >>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>>>>>> help
>> > > >>>>>>>>>>>>>>>>>>>> us
>> > > >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
>> > > >>> Jing
>> > > >>>> Ge
>> > > >>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
>> > > >>> am
>> > > >>>>>> still
>> > > >>>>>>>>>> confused
>> > > >>>>>>>>>>>>> after
>> > > >>>>>>>>>>>>>>>>>>> completely
>> > > >>>>>>>>>>>>>>>>>>>>>>> reading
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
>> > > >>>> questions:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
>> > > >>>>> registerCatalog()
>> > > >>>>>>> and
>> > > >>>>>>>>>>>>>> addCatalog()
>> > > >>>>>>>>>>>>>>>> have
>> > > >>>>>>>>>>>>>>>>>> no
>> > > >>>>>>>>>>>>>>>>>>>> big
>> > > >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
>> > > >>>> One
>> > > >>>>> of
>> > > >>>>>>>> them is
>> > > >>>>>>>>>>>>>>> responsible
>> > > >>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>> data
>> > > >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
>> > > >>>>>> persistCatalog()?
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
>> > > >>>> Map<String,
>> > > >>>>>>>> Catalog>
>> > > >>>>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>> used
>> > > >>>>>>>>>>>>>>>>>>> as a
>> > > >>>>>>>>>>>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
>> > > >>> data
>> > > >>>>>>>> persistence.
>> > > >>>>>>>>>> I
>> > > >>>>>>>>>>>>> would
>> > > >>>>>>>>>>>>>>>>> suggest
>> > > >>>>>>>>>>>>>>>>>>>>>> describing
>> > > >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
>> > > >>>>> clearly
>> > > >>>>>> in
>> > > >>>>>>>> the
>> > > >>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>> Some
>> > > >>>>>>>>>>>>>>>>> common
>> > > >>>>>>>>>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
>> > > >>>> i.e.
>> > > >>>>>> data
>> > > >>>>>>>> in
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> store
>> > > >>>>>>>>>>>>>>>>>>>>>>> should
>> > > >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
>> > > >>>> instance
>> > > >>>>>>>> should be
>> > > >>>>>>>>>>>> found
>> > > >>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> store
>> > > >>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
>> > > >>>>>> initialized
>> > > >>>>>>>> or it
>> > > >>>>>>>>>>> will
>> > > >>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>> lazy
>> > > >>>>>>>>>>>>>>>>>>>>>> initialized)
>> > > >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
>> > > >>>>>>> consistency
>> > > >>>>>>>> will
>> > > >>>>>>>>>>> be
>> > > >>>>>>>>>>>>>> taken
>> > > >>>>>>>>>>>>>>>> care
>> > > >>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>> while
>> > > >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
>> > > >>>>>> forward,
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>> option
>> > > >>>>>>>>>>>>> 2
>> > > >>>>>>>>>>>>>>>>> solution
>> > > >>>>>>>>>>>>>>>>>>>> looks
>> > > >>>>>>>>>>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
>> > > >>>>>> because,
>> > > >>>>>>>> afaiu,
>> > > >>>>>>>>>>>>> issues
>> > > >>>>>>>>>>>>>>>>>> mentioned
>> > > >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
>> > > >>>>> solved
>> > > >>>>>>>> yet. Do
>> > > >>>>>>>>>>> you
>> > > >>>>>>>>>>>>>> still
>> > > >>>>>>>>>>>>>>>>> want
>> > > >>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>> propose
>> > > >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
>> > > >>>> suggestions
>> > > >>>>>> for
>> > > >>>>>>>> both
>> > > >>>>>>>>>> of
>> > > >>>>>>>>>>>>> them?
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
>> > > >>>> there
>> > > >>>>>> are
>> > > >>>>>>>> some
>> > > >>>>>>>>>>>>>>> inconsistent
>> > > >>>>>>>>>>>>>>>>>>>>> descriptions
>> > > >>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
>> > > >>>> clean
>> > > >>>>>>> them
>> > > >>>>>>>> up?
>> > > >>>>>>>>>>>>> Thanks!
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
>> > > >>> Feng
>> > > >>>>>> Jin <
>> > > >>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
>> > > >>> and
>> > > >>>> I
>> > > >>>>>>>> completely
>> > > >>>>>>>>>>>> agree
>> > > >>>>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>>>> your
>> > > >>>>>>>>>>>>>>>>>>>> point
>> > > >>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> view.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
>> > > >>>>>> complicated
>> > > >>>>>>>> the
>> > > >>>>>>>>>>> whole
>> > > >>>>>>>>>>>>>> issue.
>> > > >>>>>>>>>>>>>>>>> First
>> > > >>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>> foremost,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
>> > > >>>> persistence
>> > > >>>>>> of
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> Catalog's
>> > > >>>>>>>>>>>>>>>>>>>> Configuration.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
>> > > >>>>>> persistence
>> > > >>>>>>>> for
>> > > >>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>> Configuration,
>> > > >>>>>>>>>>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
>> > > >>> to
>> > > >>>> the
>> > > >>>>>>>> Catalog
>> > > >>>>>>>>>>>>>> interface.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
>> > > >>> Catalog
>> > > >>>>>>>> instance
>> > > >>>>>>>>>> to a
>> > > >>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>> String>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
>> > > >>>>>>>> implementation
>> > > >>>>>>>>>> will
>> > > >>>>>>>>>>>>> throw
>> > > >>>>>>>>>>>>>>> an
>> > > >>>>>>>>>>>>>>>>>>>> exception.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> /**
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
>> > > >>> the
>> > > >>>>>>>> properties
>> > > >>>>>>>>>> of
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>>>> object.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> *
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
>> > > >>> the
>> > > >>>>>>>> properties
>> > > >>>>>>>>>> of
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>>>> object
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
>> > > >>>>>>>> UnsupportedOperationException if
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> implementing
>> > > >>>>>>>>>>>>>>>>>>>>> class
>> > > >>>>>>>>>>>>>>>>>>>>>>> does
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> not override
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> * the default
>> > > >>> implementation
>> > > >>>> of
>> > > >>>>>> this
>> > > >>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> */
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
>> > > >>>>>>>> toProperties()
>> > > >>>>>>>>>> {
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> throw new
>> > > >>>>>>>>>>>> UnsupportedOperationException("Please
>> > > >>>>>>>>>>>>>>>>> implement
>> > > >>>>>>>>>>>>>>>>>>>>>>>> toProperties
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
>> > > >>>> follows:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
>> > > >>> configured a
>> > > >>>>>>>>>> CatalogStore,
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>> toConfiguration()
>> > > >>>>>>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
>> > > >>> registering a
>> > > >>>>>>> Catalog
>> > > >>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>> > > >>>> catalogName,
>> > > >>>>>>>> Catalog
>> > > >>>>>>>>>>>>> catalog).
>> > > >>>>>>>>>>>>>>> The
>> > > >>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
>> > > >>>> Map<String,
>> > > >>>>>>>> String>
>> > > >>>>>>>>>>>>> properties
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>> saved
>> > > >>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
>> > > >>>>>> instances
>> > > >>>>>>>> do
>> > > >>>>>>>>>> not
>> > > >>>>>>>>>>>>>>> implement
>> > > >>>>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>>>> method,
>> > > >>>>>>>>>>>>>>>>>>>>>>> an
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
>> > > >>>>>>>> CatalogStore,
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> toConfiguration()
>> > > >>>>>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
>> > > >>>>>>> consistency
>> > > >>>>>>>> with
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> original
>> > > >>>>>>>>>>>>>>>>>>>> process.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
>> > > >>>>>> Catalog>
>> > > >>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>>>> at
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
>> > > >>>>>> conflicts
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
>> > > >>>>> itself
>> > > >>>>>>> and
>> > > >>>>>>>>>>> provide
>> > > >>>>>>>>>>>> a
>> > > >>>>>>>>>>>>>>>>> dedicated
>> > > >>>>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
>> > > >>> loading,
>> > > >>>>> such
>> > > >>>>>>> as
>> > > >>>>>>>>>>>>>>>>> ConfigurationCatalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> public class
>> > > >>> ConfigurationCatalog
>> > > >>>>>>>> implements
>> > > >>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>> {
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>> ConfigurationCatalog(Map<String,
>> > > >>>>>>> String>
>> > > >>>>>>>>>>>>> properties)
>> > > >>>>>>>>>>>>>> {
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
>> > > >>> the
>> > > >>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
>> > > >>> 10:03 AM
>> > > >>>>>> Shammon
>> > > >>>>>>>> FY <
>> > > >>>>>>>>>>>>>>>>>> zjureel@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
>> > > >>> the
>> > > >>>>>>>> functionality
>> > > >>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>> `CatalogStore`,
>> > > >>>>>>>>>>>>>>>>>>>>> but
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
>> > > >>>> multiple
>> > > >>>>>>>>>>>>> `registerCatalog`
>> > > >>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
>> > > >>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
>> > > >>>>>> implementation
>> > > >>>>>>> in
>> > > >>>>>>>>>> Trino
>> > > >>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>> also
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> same,
>> > > >>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
>> > > >>> only
>> > > >>>>> has
>> > > >>>>>>> one
>> > > >>>>>>>>>>>>>>> `createCatalog`,
>> > > >>>>>>>>>>>>>>>>>> which
>> > > >>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>> save
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
>> > > >>>>>> `CatalogStore`.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
>> > > >>>>>> another
>> > > >>>>>>>>>>>>>>> `registerCatalog`
>> > > >>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
>> > > >>>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
>> > > >>> `Map<String,
>> > > >>>>>>> Catalog>
>> > > >>>>>>>>>>>> catalogs`
>> > > >>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>> `CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
>> > > >>>> mentioned
>> > > >>>>>>>> before,
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> `Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
>> > > >>>>>>>> `CatalogManager`.
>> > > >>>>>>>>>> How
>> > > >>>>>>>>>>>>> about
>> > > >>>>>>>>>>>>>>>>> saving
>> > > >>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
>> > > >>> catalogs`
>> > > >>>>> and
>> > > >>>>>>>>>>>> `CatalogStore
>> > > >>>>>>>>>>>>>>>>>>> catalogStore`
>> > > >>>>>>>>>>>>>>>>>>>>>>>> together
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
>> > > >>> added
>> > > >>>> in
>> > > >>>>>>>>>>>>> `CatalogManager`?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>> > > >>> 5:22 PM
>> > > >>>>> Feng
>> > > >>>>>>>> Jin <
>> > > >>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
>> > > >>> I
>> > > >>>> am
>> > > >>>>>> very
>> > > >>>>>>>> sorry
>> > > >>>>>>>>>>> for
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>> misunderstanding
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> caused
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
>> > > >>>>> original
>> > > >>>>>>>>>>> discussion.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
>> > > >>>>>> pre-discussion
>> > > >>>>>>>> [1]
>> > > >>>>>>>>>> for
>> > > >>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>> FLIP
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
>> > > >>> such a
>> > > >>>>>>>> discussion
>> > > >>>>>>>>>>>> before.
>> > > >>>>>>>>>>>>>>>>> However,
>> > > >>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> designing
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
>> > > >>> that
>> > > >>>>> the
>> > > >>>>>>>> logic of
>> > > >>>>>>>>>>>>>>>>> CatalogManager
>> > > >>>>>>>>>>>>>>>>>>>> itself
>> > > >>>>>>>>>>>>>>>>>>>>>>>> doesn't
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
>> > > >>>> cannot
>> > > >>>>>>> only
>> > > >>>>>>>>>>> persist
>> > > >>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>> instances
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
>> > > >>>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> might
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
>> > > >>>> Catalogs,
>> > > >>>>>>>> because
>> > > >>>>>>>>>> in
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>> end
>> > > >>>>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>>>>>> still
>> > > >>>>>>>>>>>>>>>>>>>>> need
>> > > >>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> save
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
>> > > >>>>> corresponding
>> > > >>>>>> to
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>> instances.
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> I
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
>> > > >>>>>>> CatalogStore
>> > > >>>>>>>>>>>> interface
>> > > >>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
>> > > >>> this
>> > > >>>>>> part, I
>> > > >>>>>>>> also
>> > > >>>>>>>>>>> took
>> > > >>>>>>>>>>>>>>>>> inspiration
>> > > >>>>>>>>>>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Trino's
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
>> > > >>>> handling
>> > > >>>>>>>> catalog
>> > > >>>>>>>>>> with
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> same
>> > > >>>>>>>>>>>>>>>>> name
>> > > >>>>>>>>>>>>>>>>>>>>> looks a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> little
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
>> > > >>>> suggestion. I
>> > > >>>>>>> will
>> > > >>>>>>>>>>> provide
>> > > >>>>>>>>>>>> a
>> > > >>>>>>>>>>>>>>>> detailed
>> > > >>>>>>>>>>>>>>>>>>>>>> description
>> > > >>>>>>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
>> > > >>> part,
>> > > >>>>> and
>> > > >>>>>>> add
>> > > >>>>>>>> it
>> > > >>>>>>>>>> to
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
>> > > >>>>>>>> `addCatalog`
>> > > >>>>>>>>>>> still
>> > > >>>>>>>>>>>>>> looks
>> > > >>>>>>>>>>>>>>>>>> confused
>> > > >>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>> me.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
>> > > >>>>>> catalog
>> > > >>>>>>> to
>> > > >>>>>>>>>>>>>> CatalogStore,
>> > > >>>>>>>>>>>>>>>> how
>> > > >>>>>>>>>>>>>>>>>>> about
>> > > >>>>>>>>>>>>>>>>>>>>>>>> renaming
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
>> > > >>> very
>> > > >>>>>>> personal
>> > > >>>>>>>>>>>> opinion,
>> > > >>>>>>>>>>>>>> you
>> > > >>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>> decide
>> > > >>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>> take
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
>> > > >>>>> intuitive
>> > > >>>>>>> to
>> > > >>>>>>>> me.
>> > > >>>>>>>>>> I
>> > > >>>>>>>>>>>>> don't
>> > > >>>>>>>>>>>>>>> see
>> > > >>>>>>>>>>>>>>>>> any
>> > > >>>>>>>>>>>>>>>>>>>>> problem
>> > > >>>>>>>>>>>>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
>> > > >>>>> statement,
>> > > >>>>>>>> which
>> > > >>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>> called?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
>> > > >>>>>> `addCatalog`?
>> > > >>>>>>>> I'm
>> > > >>>>>>>>>>>>> wondering
>> > > >>>>>>>>>>>>>>>>> whether
>> > > >>>>>>>>>>>>>>>>>>> user
>> > > >>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>> add a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
>> > > >>>>>> stement.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
>> > > >>>> original
>> > > >>>>>>>> design
>> > > >>>>>>>>>> was
>> > > >>>>>>>>>>> to
>> > > >>>>>>>>>>>>> add
>> > > >>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>> directly
>> > > >>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
>> > > >>> would
>> > > >>>>>>> disrupt
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> existing
>> > > >>>>>>>>>>>>>>>>> logic.
>> > > >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> think
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
>> > > >>>>>>>> configuration to
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> initialize
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
>> > > >>> the
>> > > >>>>> same
>> > > >>>>>>> time
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
>> > > >>> neccessary
>> > > >>>> to
>> > > >>>>>>>> provide a
>> > > >>>>>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>>>>>> implmentation
>> > > >>>>>>>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
>> > > >>>>> `CatalogStoreFactory`?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
>> > > >>>>>> otherwise
>> > > >>>>>>> we
>> > > >>>>>>>>>> would
>> > > >>>>>>>>>>>>> need
>> > > >>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>> introduce
>> > > >>>>>>>>>>>>>>>>>>>>> an
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> additional
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
>> > > >>>> configuration
>> > > >>>>>> for
>> > > >>>>>>>> lazy
>> > > >>>>>>>>>>>>> loading.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
>> > > >>>>>>> registration
>> > > >>>>>>>> for
>> > > >>>>>>>>>>>>> catalog.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
>> > > >>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
>> > > >>>>>> interface
>> > > >>>>>>>>>> because
>> > > >>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>>>>>> an
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
>> > > >>>>>>>> initialization
>> > > >>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>> than
>> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
>> > > >>>>>>>> inaccurate. It
>> > > >>>>>>>>>>>>> should
>> > > >>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>> lazy
>> > > >>>>>>>>>>>>>>>>>>>>>>> registration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
>> > > >>>>>>>> registration. I
>> > > >>>>>>>>>>> have
>> > > >>>>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>>>> updated
>> > > >>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> title
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>> > > >>>> 4:27 PM
>> > > >>>>>>>> Shammon
>> > > >>>>>>>>>> FY <
>> > > >>>>>>>>>>>>>>>>>>> zjureel@gmail.com
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
>> > > >>>>>>>> `registerCatalog`
>> > > >>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
>> > > >>>>> method
>> > > >>>>>>>> named
>> > > >>>>>>>>>>>>>> `addCatalog`
>> > > >>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
>> > > >>> the
>> > > >>>>>> memory
>> > > >>>>>>>>>> catalog
>> > > >>>>>>>>>>>> is a
>> > > >>>>>>>>>>>>>>>>> `cache`,
>> > > >>>>>>>>>>>>>>>>>> I
>> > > >>>>>>>>>>>>>>>>>>>>> think
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
>> > > >>>>>> exposed
>> > > >>>>>>> to
>> > > >>>>>>>>>>> users.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
>> > > >>>>>> pre-discussion
>> > > >>>>>>>> [1]
>> > > >>>>>>>>>> for
>> > > >>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>> Please
>> > > >>>>>>>>>>>>>>>>>>>>>>> correct
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> me
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> if
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
>> > > >>>>> conclusion
>> > > >>>>>>> of
>> > > >>>>>>>> that
>> > > >>>>>>>>>>>>>>> discussion
>> > > >>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>> use
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
>> > > >>>>>> interface
>> > > >>>>>>>> and
>> > > >>>>>>>>>>>>> implement
>> > > >>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>> different
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> stores
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> such
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
>> > > >>>> external
>> > > >>>>>>>> system.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
>> > > >>>>> between
>> > > >>>>>>> the
>> > > >>>>>>>>>>> current
>> > > >>>>>>>>>>>>> FLIP
>> > > >>>>>>>>>>>>>>>>> design
>> > > >>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
>> > > >>> of
>> > > >>>>> the
>> > > >>>>>>>>>> discussion
>> > > >>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>> thread
>> > > >>>>>>>>>>>>>>>>>> [1] ?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>
>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>> > > >>>>> 3:41 PM
>> > > >>>>>>>> yuxia <
>> > > >>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
>> > > >>> driving
>> > > >>>>> this
>> > > >>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
>> > > >>>>> handling
>> > > >>>>>>>> catalog
>> > > >>>>>>>>>>> with
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> same
>> > > >>>>>>>>>>>>>>>>>> name
>> > > >>>>>>>>>>>>>>>>>>>>>> looks a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> little
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
>> > > >>>>> it'll
>> > > >>>>>> be
>> > > >>>>>>>>>> better
>> > > >>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> explain
>> > > >>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> java
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> doc
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
>> > > >>> give a
>> > > >>>>>> brief
>> > > >>>>>>>>>> example
>> > > >>>>>>>>>>> in
>> > > >>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
>> > > >>> name
>> > > >>>>>>>> `addCatalog`
>> > > >>>>>>>>>>>> still
>> > > >>>>>>>>>>>>>>> looks
>> > > >>>>>>>>>>>>>>>>>>> confused
>> > > >>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>> me.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
>> > > >>>> storing
>> > > >>>>>>>> catalog to
>> > > >>>>>>>>>>>>>>>> CatalogStore,
>> > > >>>>>>>>>>>>>>>>>> how
>> > > >>>>>>>>>>>>>>>>>>>>> about
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
>> > > >>> It's
>> > > >>>>> very
>> > > >>>>>>>> personal
>> > > >>>>>>>>>>>>>> opinion,
>> > > >>>>>>>>>>>>>>>> you
>> > > >>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>> decide
>> > > >>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> take
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
>> > > >>>>>>> statement,
>> > > >>>>>>>>>> which
>> > > >>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>> called?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
>> > > >>>>>>>> `addCatalog`?
>> > > >>>>>>>>>> I'm
>> > > >>>>>>>>>>>>>>> wondering
>> > > >>>>>>>>>>>>>>>>>>> whether
>> > > >>>>>>>>>>>>>>>>>>>>> user
>> > > >>>>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> add a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
>> > > >>> SQL
>> > > >>>>>>>> stement.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
>> > > >>>> neccessary
>> > > >>>>> to
>> > > >>>>>>>>>> provide a
>> > > >>>>>>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>>>>>>> implmentation
>> > > >>>>>>>>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
>> > > >>>>>>> `CatalogStoreFactory`?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
>> > > >>>>>>>> registration
>> > > >>>>>>>>>> for
>> > > >>>>>>>>>>>>>> catalog.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
>> > > >>> catalog
>> > > >>>>>> with
>> > > >>>>>>>> CREATE
>> > > >>>>>>>>>>>>>> CATALOG,
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
>> > > >>>> used
>> > > >>>>>> by
>> > > >>>>>>>>>> default.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
>> > > >>>>> registration
>> > > >>>>>> is
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>>> behavior,
>> > > >>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>> there
>> > > >>>>>>>>>>>>>>>>>>>>>>>> any
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> way
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
>> > > >>> to
>> > > >>>>>>>> synchronous
>> > > >>>>>>>>>>>>>>> registration
>> > > >>>>>>>>>>>>>>>>> just
>> > > >>>>>>>>>>>>>>>>>>>> like
>> > > >>>>>>>>>>>>>>>>>>>>>>>> before?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
>> > > >>>>> `addCatalog`
>> > > >>>>>>> and
>> > > >>>>>>>>>>>>>>>> `registerCatalog`
>> > > >>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
>> > > >>>>>>>> registration,
>> > > >>>>>>>>>> it
>> > > >>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>> well
>> > > >>>>>>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>>>> CREATE
>> > > >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
>> > > >>>> but
>> > > >>>>>> then
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> following
>> > > >>>>>>>>>>>>>>>>> CREATE
>> > > >>>>>>>>>>>>>>>>>>>> TABLE
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> statement
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
>> > > >>> fail
>> > > >>>>> to
>> > > >>>>>>>> open.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
>> > > >>>> change
>> > > >>>>>>> which
>> > > >>>>>>>>>>> should
>> > > >>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>> highlighted
>> > > >>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
>> > > >>> part.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
>> > > >>>> asynchronous
>> > > >>>>>>>>>>>> registration, I
>> > > >>>>>>>>>>>>>>> would
>> > > >>>>>>>>>>>>>>>>>> like
>> > > >>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>> expect
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> there
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
>> > > >>>> open
>> > > >>>>>> or
>> > > >>>>>>>>>> register
>> > > >>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> background,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> but
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
>> > > >>> comments,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
>> > > >>>> will
>> > > >>>>> be
>> > > >>>>>>>>>>> initialized
>> > > >>>>>>>>>>>>> if
>> > > >>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>> has
>> > > >>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>> been
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
>> > > >>>> initialization
>> > > >>>>>>>> process
>> > > >>>>>>>>>>> fails,
>> > > >>>>>>>>>>>>>> these
>> > > >>>>>>>>>>>>>>>>>>>> statements
>> > > >>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
>> > > >>> successfully."
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
>> > > >>> lazy
>> > > >>>>>>>>>> initialization
>> > > >>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>> than
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
>> > > >>>>>>>>>> jinfeng1995@gmail.com
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
>> > > >>>>>>> dev@flink.apache.org
>> > > >>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
>> > > >>> 03日
>> > > >>>> 下午
>> > > >>>>>>>> 3:27:45
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
>> > > >>>> 295:
>> > > >>>>>>>> Support
>> > > >>>>>>>>>>>>>> persistence
>> > > >>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
>> > > >>>>> registration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
>> > > >>>> all
>> > > >>>>>> for
>> > > >>>>>>>> your
>> > > >>>>>>>>>>>>> interest
>> > > >>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>>>> DISCUSS.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
>> > > >>> catalog
>> > > >>>>>> with
>> > > >>>>>>>> the
>> > > >>>>>>>>>> same
>> > > >>>>>>>>>>>>> name
>> > > >>>>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>>> exists
>> > > >>>>>>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>>>> both
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
>> > > >>>> crucial
>> > > >>>>>>>> point.
>> > > >>>>>>>>>>> Based
>> > > >>>>>>>>>>>> on
>> > > >>>>>>>>>>>>>> my
>> > > >>>>>>>>>>>>>>>>>> current
>> > > >>>>>>>>>>>>>>>>>>>>>> personal
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
>> > > >>>>>> Map<String,
>> > > >>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>> serve
>> > > >>>>>>>>>>>>>>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>>>>>>>> a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
>> > > >>> catalogs
>> > > >>>>> and
>> > > >>>>>>>> have
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>> highest
>> > > >>>>>>>>>>>>>>>>>>> priority.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
>> > > >>> methods
>> > > >>>>> that
>> > > >>>>>>> can
>> > > >>>>>>>>>> affect
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
>> > > >>> registerCatalog(String
>> > > >>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
>> > > >>>>> catalog
>> > > >>>>>>>> instance
>> > > >>>>>>>>>>>> into
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
>> > > >>>> unregisterCatalog(String
>> > > >>>>>>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
>> > > >>> the
>> > > >>>>>> catalog
>> > > >>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>> corresponding
>> > > >>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
>> > > >>>>> Catalog>
>> > > >>>>>>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
>> > > >>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
>> > > >>>> retrieves
>> > > >>>>>> the
>> > > >>>>>>>>>>>>> corresponding
>> > > >>>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
>> > > >>> Catalog>
>> > > >>>>>>>> catalogs. If
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>> does
>> > > >>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>>>> exist,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
>> > > >>>> corresponding
>> > > >>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
>> > > >>> puts
>> > > >>>>> the
>> > > >>>>>>>>>>> initialized
>> > > >>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>>>>>> into
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
>> > > >>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
>> > > >>> methods
>> > > >>>>>> only
>> > > >>>>>>>> modify
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
>> > > >>>>>>> catalogName,
>> > > >>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>> String>
>> > > >>>>>>>>>>>>>>>>>>>>>>> properties)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
>> > > >>>>>>> properties
>> > > >>>>>>>> to
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>> catalogStore
>> > > >>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>> checks
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
>> > > >>>>>> catalogName
>> > > >>>>>>>> with
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>> same
>> > > >>>>>>>>>>>>>>>> name.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
>> > > >>> removeCatalog(String
>> > > >>>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
>> > > >>> the
>> > > >>>>>>> specified
>> > > >>>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> specified
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
>> > > >>>>>>> catalogStore.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
>> > > >>>> possible
>> > > >>>>>>>> conflict
>> > > >>>>>>>>>>>>>> scenarios:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
>> > > >>> corresponding
>> > > >>>>>>>> catalogName
>> > > >>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>> exists
>> > > >>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
>> > > >>> in
>> > > >>>> the
>> > > >>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>> Catalog>,
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>> > > >>>>>>>> catalogName,
>> > > >>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
>> > > >>> directly
>> > > >>>>> saved
>> > > >>>>>>> to
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
>> > > >>> corresponding
>> > > >>>>>>>> catalogName
>> > > >>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>> exists
>> > > >>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>> both
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
>> > > >>>>>>> Map<String,
>> > > >>>>>>>>>>>> Catalog>,
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
>> > > >>>>> catalog)
>> > > >>>>>>>> method
>> > > >>>>>>>>>>> will
>> > > >>>>>>>>>>>>>> fail.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
>> > > >>> corresponding
>> > > >>>>>>>> catalogName
>> > > >>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>> exists
>> > > >>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
>> > > >>> the
>> > > >>>>>>>>>>>> addCatalog(String
>> > > >>>>>>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
>> > > >>>> method
>> > > >>>>>> can
>> > > >>>>>>>>>> directly
>> > > >>>>>>>>>>>>> save
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> properties
>> > > >>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
>> > > >>>>>>>>>> getCatalog(String
>> > > >>>>>>>>>>>>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
>> > > >>>> for
>> > > >>>>>>>>>>> initialization
>> > > >>>>>>>>>>>>>>> because
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> corresponding
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
>> > > >>> already
>> > > >>>>>> exists
>> > > >>>>>>>> in
>> > > >>>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
>> > > >>>>>>>>>>>>> unregisterCatalog(String
>> > > >>>>>>>>>>>>>>>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
>> > > >>>>>>>> corresponding to
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> original
>> > > >>>>>>>>>>>>>>>>>>>>> catalogName
>> > > >>>>>>>>>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
>> > > >>> confuse
>> > > >>>>>> users
>> > > >>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
>> > > >>>>> catalog)`
>> > > >>>>>>> in
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>>>> `Map<String,
>> > > >>>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>> `registerCatalog(String
>> > > >>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>> String>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> properties)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
>> > > >>>> lead
>> > > >>>>>> to
>> > > >>>>>>>>>>>> confusion. I
>> > > >>>>>>>>>>>>>>>> suggest
>> > > >>>>>>>>>>>>>>>>>>>>> changing
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
>> > > >>> to
>> > > >>>>>>>>>>> addCatalog(String
>> > > >>>>>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
>> > > >>> as
>> > > >>>>>>>> previously
>> > > >>>>>>>>>>>>> mentioned
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
>> > > >>>>> `registerCatalog(String
>> > > >>>>>>>>>>>>>>> catalogName,Catalog
>> > > >>>>>>>>>>>>>>>>>>>> catalog,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
>> > > >>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
>> > > >>>> already
>> > > >>>>> an
>> > > >>>>>>>>>> instance,
>> > > >>>>>>>>>>>>>> adding
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>> "lazyInit"
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
>> > > >>>>>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
>> > > >>>> necessarily
>> > > >>>>>>>> result in
>> > > >>>>>>>>>>> lazy
>> > > >>>>>>>>>>>>>>>>>>> initialization.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
>> > > >>>> about
>> > > >>>>>>>>>> encryption
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
>> > > >>>>>>> necessary,
>> > > >>>>>>>> but
>> > > >>>>>>>>>>>>> perhaps
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> encryption
>> > > >>>>>>>>>>>>>>>>>>>>>>> logic
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
>> > > >>> in
>> > > >>>>> the
>> > > >>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>> when
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> data
>> > > >>>>>>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>>>>>> actually
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
>> > > >>> the
>> > > >>>>>>>>>>> FileCatalogStore
>> > > >>>>>>>>>>>>>> could
>> > > >>>>>>>>>>>>>>>>>> encrypt
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> data
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> when
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
>> > > >>>>> while
>> > > >>>>>>> the
>> > > >>>>>>>>>>>>>>>> MemoryCatalogStore
>> > > >>>>>>>>>>>>>>>>>>> does
>> > > >>>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> require
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
>> > > >>> the
>> > > >>>>>>>>>>>>>> `MemoryCatalogStore`?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
>> > > >>> necessary
>> > > >>>> to
>> > > >>>>>>> have a
>> > > >>>>>>>>>>>>>>>>> MemoryCatalogStore
>> > > >>>>>>>>>>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
>> > > >>>> saves
>> > > >>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>> configurations
>> > > >>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>> memory.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
>> > > >>> to
>> > > >>>>>>>> implement
>> > > >>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>> loading
>> > > >>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
>> > > >>> introduce
>> > > >>>>>>>> additional
>> > > >>>>>>>>>>> cache.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
>> > > >>>>>>>> registration
>> > > >>>>>>>>>>>> failed?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
>> > > >>> critical
>> > > >>>>>>> concern.
>> > > >>>>>>>>>> When
>> > > >>>>>>>>>>>>>>> executing
>> > > >>>>>>>>>>>>>>>>> DDL,
>> > > >>>>>>>>>>>>>>>>>>>> DQL,
>> > > >>>>>>>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>> DML
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
>> > > >>>> reference a
>> > > >>>>>>>> specific
>> > > >>>>>>>>>>>>> Catalog,
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
>> > > >>> if it
>> > > >>>>> has
>> > > >>>>>>> not
>> > > >>>>>>>>>> been
>> > > >>>>>>>>>>>>>>>> initialized
>> > > >>>>>>>>>>>>>>>>>> yet.
>> > > >>>>>>>>>>>>>>>>>>>> If
>> > > >>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
>> > > >>>>> fails,
>> > > >>>>>>>> these
>> > > >>>>>>>>>>>>>> statements
>> > > >>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>> executed
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
>> > > >>>>>> Catalog>
>> > > >>>>>>>>>> catalogs
>> > > >>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
>> > > >>>> have
>> > > >>>>>> the
>> > > >>>>>>>> same
>> > > >>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>> name,
>> > > >>>>>>>>>>>>>>>>>> but
>> > > >>>>>>>>>>>>>>>>>>>>>>>> correponding
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
>> > > >>>>> instances.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
>> > > >>>> resolved
>> > > >>>>> by
>> > > >>>>>>>>>> referring
>> > > >>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> previous
>> > > >>>>>>>>>>>>>>>>>>>>>>>> responses.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
>> > > >>> the
>> > > >>>>>>> Catalog
>> > > >>>>>>>>>> that
>> > > >>>>>>>>>>>> has
>> > > >>>>>>>>>>>>>> been
>> > > >>>>>>>>>>>>>>>>> most
>> > > >>>>>>>>>>>>>>>>>>>>> recently
>> > > >>>>>>>>>>>>>>>>>>>>>>>> used
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
>> > > >>>> for
>> > > >>>>>>>> subsequent
>> > > >>>>>>>>>>>> use.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
>> > > >>>>>> 10:58 PM
>> > > >>>>>>>>>> Xianxun
>> > > >>>>>>>>>>>> Ye
>> > > >>>>>>>>>>>>> <
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
>> > > >>> this
>> > > >>>>>> Flip,
>> > > >>>>>>>> I do
>> > > >>>>>>>>>>>>> believe
>> > > >>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>> Flip
>> > > >>>>>>>>>>>>>>>>>>>>> could
>> > > >>>>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
>> > > >>>> work
>> > > >>>>>> for
>> > > >>>>>>>> also
>> > > >>>>>>>>>>>>> manages a
>> > > >>>>>>>>>>>>>>> lot
>> > > >>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>> catalogs,
>> > > >>>>>>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
>> > > >>>> becomes
>> > > >>>>>>> slow
>> > > >>>>>>>>>>> because
>> > > >>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>> loading a
>> > > >>>>>>>>>>>>>>>>>>>>> number
>> > > >>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
>> > > >>> catalogs in
>> > > >>>>> the
>> > > >>>>>>>> user's
>> > > >>>>>>>>>>>> Flink
>> > > >>>>>>>>>>>>>> SQL
>> > > >>>>>>>>>>>>>>>>>> through
>> > > >>>>>>>>>>>>>>>>>>>>>> regular
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
>> > > >>>>> catalogs
>> > > >>>>>>> to
>> > > >>>>>>>>>>> improve
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> speed
>> > > >>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>> task
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
>> > > >>> flip-295, I
>> > > >>>>>> have
>> > > >>>>>>>> some
>> > > >>>>>>>>>>>>>> questions:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
>> > > >>>>> catalog
>> > > >>>>>>> with
>> > > >>>>>>>>>>> CREATE
>> > > >>>>>>>>>>>>>>> CATALOG,
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
>> > > >>>> used
>> > > >>>>>> by
>> > > >>>>>>>>>> default.
>> > > >>>>>>>>>>>>> What
>> > > >>>>>>>>>>>>>> if
>> > > >>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
>> > > >>>>> implement
>> > > >>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>> registration?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
>> > > >>> same
>> > > >>>>>>>> question
>> > > >>>>>>>>>> with
>> > > >>>>>>>>>>>>>>> Shammon’s
>> > > >>>>>>>>>>>>>>>>>> first
>> > > >>>>>>>>>>>>>>>>>>>>>>> comment.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> The
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
>> > > >>>>> catalogs
>> > > >>>>>>> and
>> > > >>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>> catalogstore
>> > > >>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
>> > > >>> have
>> > > >>>> the
>> > > >>>>>>> same
>> > > >>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>> name,
>> > > >>>>>>>>>>>>>>>> but
>> > > >>>>>>>>>>>>>>>>>>>>>> correponding
>> > > >>>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
>> > > >>>>> instances.
>> > > >>>>>>> For
>> > > >>>>>>>>>>> example:
>> > > >>>>>>>>>>>>>>>>> catalog1->
>> > > >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
>> > > >>> catalog1 ->
>> > > >>>>>>>> hiveCatalog
>> > > >>>>>>>>>>> in
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> catalogstore.
>> > > >>>>>>>>>>>>>>>>>>>>>>> Would
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
>> > > >>> should
>> > > >>>>>> define
>> > > >>>>>>>>>>> `catalogs`
>> > > >>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>> `catalogstore`
>> > > >>>>>>>>>>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
>> > > >>> 11:13，Hang
>> > > >>>>>> Ruan
>> > > >>>>>>> <
>> > > >>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> 写道：
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
hi, Hang
Thanks for your reply, Make sense to me, I updated the FLIP.
Best,
Feng
On Fri, Jun 9, 2023 at 3:34 PM Hang Ruan <ru...@gmail.com> wrote:
> Hi, Feng.
>
> I reviewed this FLIP again and found some little places that may need to
> optimize.
>
> 1. `CatalogDescriptor` may need a private constructor.
> 2. The method `initCatalog` in `CatalogManager` is not introduced.
>
> Best,
> Hang
>
> Feng Jin <ji...@gmail.com> 于2023年6月6日周二 21:17写道：
>
> > Hi Leonard,
> >
> > Thanks for your reply.
> >
> > > 1. a How to construct a CatalogDescriptor ?
> >
> > I think it would be helpful to add a method for constructing a
> > CatalogDescriptor, as you mentioned in 1.c. I will update the
> documentation
> > later.
> >
> > > 1.b How to visit the fields ? Could we use Configuration instead of
> > Map<String, String> ?
> >
> > I believe that the use of Map<String, String> options is only intended
> for
> > creating a catalog and not for accessing internal parameters.
> >
> > Since all of the relevant parameters for CREATE CATALOG are also stored
> in
> > Map<String, String> options, my understanding is that using Map<String,
> > String> options should suffice.
> >
> > Here is the implementation of execute CREATE CATALOG statement.
> > ```java
> > private TableResultInternal createCatalog(CreateCatalogOperation
> operation)
> > {
> > String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
> > try {
> > String catalogName = operation.getCatalogName();
> > Map<String, String> properties = operation.getProperties();
> >
> > Catalog catalog =
> > FactoryUtil.createCatalog(
> > catalogName,
> > properties,
> > tableConfig,
> > resourceManager.getUserClassLoader());
> > catalogManager.registerCatalog(catalogName, catalog);
> >
> > return TableResultImpl.TABLE_RESULT_OK;
> > } catch (CatalogException e) {
> > throw new ValidationException(exMsg, e);
> > }
> > }
> > ```
> >
> >
> > > 2. Do we have plan to offer a default CatalogStore if user didn’t
> config
> > this?
> >
> > Yes, the in-memory catalogStore will be used as the default CatalogStore
> > even if the user has not configured one
> >
> >
> > Best,
> > Feng
> >
> >
> > On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xb...@gmail.com> wrote:
> >
> > > Hi, Feng
> > >
> > > Sorry for reply late, but I’ve some comments about the FLIP
> > >
> > >
> > > 1. The introduced Public class CatalogDescriptor seems missed some
> > > necessary component
> > > a) How to construct a CatalogDescriptor ?
> > > b) How to visit the fields ? Could we use Configuration instead of
> > > Map<String, String> ?
> > > c) Could we offer a built-in factory method to build a
> > CatalogDescriptor
> > > like
> > > public static CatalogDescriptor of(String catalogName,
> Configuration
> > > configuration)
> > >
> > > 2. The FLIP said “By default, there are two built-in CatalogStores
> > > available: the In-Memory CatalogStore and the File CatalogStore” ,
> > > Do we have plan to offer a default CatalogStore if user didn’t config
> > > this? IIUC, users can obtains the benefits from lazy catalog
> > > initialization If
> > > we have a default catalogstore even it is in-memory.
> > >
> > > Best,
> > > Leonard
> > >
> > >
> > >
> > > > On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > Thank you everyone for your valuable input. If there are no further
> > > > questions or concerns about the FLIP[1], I would like to start voting
> > > > tomorrow (6/7).
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > >
> > > > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com>
> wrote:
> > > >
> > > >> Hi Samrat,
> > > >>
> > > >> Thanks for your advice.
> > > >>
> > > >>> 1. The createCatalog method does not mention any exceptions being
> > > >> thrown.
> > > >>
> > > >> CreateCatalog will throw CatalogException like registerCatalog. As
> > > >> CatalogException is a RuntimeException,
> > > >> there is no explicit declaration of throwing Exceptions in
> > > CatalogManager
> > > >> and TableEnvironment.
> > > >> To avoid misunderstandings, I have added the "throw
> CatalogException"
> > > flag
> > > >> to the createCatalog method definition of CatalogStore.
> > > >>
> > > >>> 2. Could you please provide an exhaustive list of the supported
> > kinds?
> > > >>
> > > >> Sure, the documentation now includes the configuration of the
> > built-in
> > > >> CatalogStore as well as how to configure a custom CatalogStore.
> > > >>
> > > >>
> > > >> Best,
> > > >> Feng
> > > >>
> > > >>
> > > >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi Feng,
> > > >>>
> > > >>> Thank you for providing the proposal. I believe this feature will
> be
> > > >>> highly
> > > >>> valuable.
> > > >>>
> > > >>> I have a couple of inquiries:
> > > >>>
> > > >>> 1. According to the documentation [1], the createCatalog method
> does
> > > not
> > > >>> mention any exceptions being thrown. However, I would like to
> confirm
> > > if
> > > >>> it
> > > >>> is always true that there will be no failures in all scenarios.
> > Please
> > > let
> > > >>> me know if there is any additional information I may have missed.
> > > >>>
> > > >>> 2. Regarding the registration process using the
> > > `table.catalog-store.kind`
> > > >>> configuration, could you please provide an exhaustive list of the
> > > >>> supported
> > > >>> kinds?
> > > >>> It would be great to have a comprehensive understanding of the
> > > options
> > > >>> available.
> > > >>>
> > > >>> Bests,
> > > >>> Samrat
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >>>
> > > >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com>
> > wrote:
> > > >>>
> > > >>>> Hi Hang and Jark
> > > >>>>
> > > >>>> Thank you very much for your reply.
> > > >>>>
> > > >>>> @Jark
> > > >>>>> 1. Could you move the CatalogStore registration API to the
> "Public
> > > >>>> Interface" section?
> > > >>>>
> > > >>>> Indeed, it is more reasonable.
> > > >>>>
> > > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> > > >>>>
> > > >>>> Sure, the names that you provided are indeed more logical and
> > easier
> > > >>> to
> > > >>>> understand.
> > > >>>>
> > > >>>>> 3. About the open/close method in CatalogStoreFactory.
> > > >>>>
> > > >>>> I also agree with the proposed requirement scenario and design. I
> > have
> > > >>>> already made modifications to the interface.
> > > >>>>
> > > >>>> @Hang
> > > >>>>
> > > >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> > > close
> > > >>>> the resource.
> > > >>>>
> > > >>>> I have already added the missing method.
> > > >>>>
> > > >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > > >>>>
> > > >>>> Sorry for the misspelling, I updated this part.
> > > >>>>
> > > >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
> > > methods
> > > >>>> to init or close its resource
> > > >>>>
> > > >>>> This is indeed a scenario that needs to be considered, and I have
> > > added
> > > >>>> relevant methods accordingly.
> > > >>>> Additionally, I have included some relevant description
> information
> > in
> > > >>> the
> > > >>>> documentation to help others understand
> > > >>>> why the open/close methods need to be added.
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>> Feng
> > > >>>>
> > > >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> > > >>>>
> > > >>>>> Hi Jing,
> > > >>>>>
> > > >>>>> Thank you for the update.
> > > >>>>>
> > > >>>>> 1. Could you move the CatalogStore registration API to the
> "Public
> > > >>>>> Interface" section?
> > > >>>>> "Proposed Changes" is more like a place to describe the
> > > implementation
> > > >>>>> details.
> > > >>>>>
> > > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> > > >>> Besides,
> > > >>>>> the config key
> > > >>>>> name should be hierarchical[1]. Therefore, it may be better to
> use:
> > > >>>>> "table.catalog-store.kind"
> > > >>>>> "table.catalog-store.file.path"
> > > >>>>>
> > > >>>>> 3. I think Hang's suggestions make sense.
> > > >>>>>
> > > >>>>> Others look good to me.
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Jark
> > > >>>>>
> > > >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
> > > >>> wrote:
> > > >>>>>
> > > >>>>>> Hi, Feng.
> > > >>>>>>
> > > >>>>>> Thanks for the update.
> > > >>>>>> The current design looks good to me. I have some minor comments.
> > > >>>>>>
> > > >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> > > >>> close
> > > >>>>> the
> > > >>>>>> resource. For example, when storing the information in MySQL,
> the
> > > >>> store
> > > >>>>>> needs to open and close the connections.
> > > >>>>>>
> > > >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > > >>>>>>
> > > >>>>>> 3. About the usage in the sql gateway.
> > > >>>>>> Considering the usage in sql gateway, the sql gateway may
> create a
> > > >>>>>> CatalogStore for each session.
> > > >>>>>> If we are using the MySqlCatalogStore, there would be so many
> > > >>>>> connections.
> > > >>>>>> How can we reuse the connection among these sessions?
> > > >>>>>> I think sql gateway need to maintain a connection pool in
> > > >>>>>> the CatalogStoreFactory and each session get its own connection
> > from
> > > >>>> the
> > > >>>>>> pool when it is created.
> > > >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close`
> methods
> > > >>> to
> > > >>>>> init
> > > >>>>>> or close its resource.
> > > >>>>>> Or is there a better way?
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Hang
> > > >>>>>>
> > > >>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> > > >>>>>>
> > > >>>>>>> Thanks Jingsong.
> > > >>>>>>>
> > > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
> > > >>>>>>>
> > > >>>>>>> +1 For this, I have already updated FLIP.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Feng
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> > > >>> jingsonglee0@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Thanks Feng,
> > > >>>>>>>>
> > > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > >>>>>>>> functions are converted to createxxx from registerxxx.
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <
> jinfeng1995@gmail.com
> > > >>>>
> > > >>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> Hi jark, thanks for your suggestion.
> > > >>>>>>>>>
> > > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> > > >>> the
> > > >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
> > > >>>>> Otherwise,
> > > >>>>>>>> there
> > > >>>>>>>>> might be some data inconsistencies when CatalogStore is
> > > >>> changed.
> > > >>>>>>>>>
> > > >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
> > > >>>>>> TableEnv.
> > > >>>>>>>>> Therefore, my current proposal is to add a method to
> configure
> > > >>>> the
> > > >>>>>>>>> CatalogStore in EnvironmentSettings.
> > > >>>>>>>>>
> > > >>>>>>>>> // Initialize a catalog Store
> > > >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
> > > >>>>>>>>>
> > > >>>>>>>>> // set up the Table API
> > > >>>>>>>>> final EnvironmentSettings settings =
> > > >>>>>>>>>
> > > >>> EnvironmentSettings.newInstance().inBatchMode()
> > > >>>>>>>>> .withCatalogStore(catalogStore)
> > > >>>>>>>>> .build();
> > > >>>>>>>>>
> > > >>>>>>>>> final TableEnvironment tableEnv =
> > > >>>>>>>> TableEnvironment.create(settings);
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> > > >>>> default
> > > >>>>>>>> method,
> > > >>>>>>>>> not an interface method?
> > > >>>>>>>>>
> > > >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
> > > >>> that
> > > >>>> as
> > > >>>>> a
> > > >>>>>>> new
> > > >>>>>>>>> interface, the CatalogStoreFactory should not have a default
> > > >>>> method
> > > >>>>>> but
> > > >>>>>>>> an
> > > >>>>>>>>> interface method. I have already modified the interface.
> > > >>>>>>>>>
> > > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> > > >>>>>>>>> deprecated`registerCatalog`.
> > > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> > > >>> mention
> > > >>>> the
> > > >>>>>>>> changed
> > > >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> > > >>>> other
> > > >>>>>>>>> persisted catalog store) is used.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> [1].
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Best,
> > > >>>>>>>>> Feng
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
> > > >>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>
> > > >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
> > > >>>>>>>>>> The current design looks pretty good to me. I just have some
> > > >>>>> minor
> > > >>>>>>>>>> comments.
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> > > >>> the
> > > >>>>>>>> CatalogStore
> > > >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
> > > >>> there
> > > >>>>>> might
> > > >>>>>>> be
> > > >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
> > > >>>>>>>>>>
> > > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> > > >>>> default
> > > >>>>>>>> method,
> > > >>>>>>>>>> not an interface method?
> > > >>>>>>>>>>
> > > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> > > >>>>> deprecated
> > > >>>>>>>>>> `registerCatalog`.
> > > >>>>>>>>>>
> > > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> > > >>> mention
> > > >>>> the
> > > >>>>>>>> changed
> > > >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> > > >>>>> other
> > > >>>>>>>>>> persisted catalog store) is used.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best,
> > > >>>>>>>>>> Jark
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
> > > >>> jinfeng1995@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> @Ron
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
> > > >>>> method,
> > > >>>>>> do
> > > >>>>>>> we
> > > >>>>>>>>>> need
> > > >>>>>>>>>>> to provide a default implementation?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Yes, we will provide a default
> > > >>> InMemoryCatalogStoreFactory to
> > > >>>>>>> create
> > > >>>>>>>> an
> > > >>>>>>>>>>> InMemoryCatalogStore.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
> > > >>> initializing
> > > >>>>> it,
> > > >>>>>>>> whether
> > > >>>>>>>>>> we
> > > >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Yes, in the current design, catalogs are stored as
> > > >>> snapshots,
> > > >>>>> and
> > > >>>>>>>> once
> > > >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
> > > >>>>>> Catalog>
> > > >>>>>>>>>>> catalogs.
> > > >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
> > > >>> the
> > > >>>>>> primary
> > > >>>>>>>>>> source
> > > >>>>>>>>>>> for obtaining the corresponding Catalog.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
> > > >>>>>>>>>>> `catalog.store.path`?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
> > > >>> seem a
> > > >>>>> bit
> > > >>>>>>>>>> strange. I
> > > >>>>>>>>>>> will update the FLIP.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> @Shammon
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thank you for the review. I have made the necessary
> > > >>>>> corrections.
> > > >>>>>>>>>>> Regarding the modifications made to the Public Interface,
> > > >>> I
> > > >>>>> have
> > > >>>>>>> also
> > > >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best,
> > > >>>>>>>>>>> Feng
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> > > >>>> zjureel@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi feng,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for updating, I have some minor comments
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
> > > >>>>>> `Public
> > > >>>>>>>>>>>> Interfaces`, it is not a public interface.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
> > > >>> and
> > > >>>>>>>>>>>> `CatalogStoreFactory`
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
> > > >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
> > > >>>>> should
> > > >>>>>> be
> > > >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
> > > >>>>>>>>>>>> catalogStore.get(catalogName);`
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> Shammon FY
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
> > > >>>> ron9.liu@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi, Feng
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
> > > >>>> useful
> > > >>>>>> for
> > > >>>>>>>>>> catalog
> > > >>>>>>>>>>>>> management.
> > > >>>>>>>>>>>>> I have some small questions:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> 1. Regarding the
> > > >>> CatalogStoreFactory#createCatalogStore
> > > >>>>>> method,
> > > >>>>>>>> do we
> > > >>>>>>>>>>>> need
> > > >>>>>>>>>>>>> to provide a default implementation?
> > > >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
> > > >>>> initializing
> > > >>>>>> it,
> > > >>>>>>>>>> whether
> > > >>>>>>>>>>> we
> > > >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
> > > >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
> > > >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
> > > >>> them to
> > > >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>> Ron
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
> > > >>> 21:19写道：
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi yuxia
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
> > > >>>>> register
> > > >>>>>>> the
> > > >>>>>>>>>>> Catalog,
> > > >>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>> initialize it and open it. right?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
> > > >>>>>> original
> > > >>>>>>>>>> CREATE
> > > >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
> > > >>>>>>>>>> registerCatalog
> > > >>>>>>>>>>> so
> > > >>>>>>>>>>>>>> that parameter validation can be performed.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
> > > >>>>>>> reflected
> > > >>>>>>>> in
> > > >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
> > > >>>>> catalog
> > > >>>>>>>>>>>>> configurations,
> > > >>>>>>>>>>>>>> only initialization is required in getCatalog.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > >>>>>>>> luoyuxia@alumni.sjtu.edu.cn
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi, Feng.
> > > >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
> > > >>>>>>>> initialization*.
> > > >>>>>>>>>>> If
> > > >>>>>>>>>>>>> i'm
> > > >>>>>>>>>>>>>>> wrong, please correct me.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
> > > >>>> access
> > > >>>>>> the
> > > >>>>>>>>>>> catalog,
> > > >>>>>>>>>>>>> then
> > > >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
> > > >>>>> Changes,
> > > >>>>>>>> once we
> > > >>>>>>>>>>>>>> register
> > > >>>>>>>>>>>>>>> the Catalog,
> > > >>>>>>>>>>>>>>> we initialize it and open it. right?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>> Yuxia
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> ----- 原始邮件 -----
> > > >>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > >>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
> > > >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > > >>>>>> Catalog
> > > >>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>> and asynchronous registration
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> One of the major changes is that current design
> > > >>> will
> > > >>>>>>> provide
> > > >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
> > > >>>> of a
> > > >>>>>>>> cache,
> > > >>>>>>>>>>> which
> > > >>>>>>>>>>>>>> means
> > > >>>>>>>>>>>>>>> once it has been initialized, any changes done by
> > > >>>> other
> > > >>>>>>>> sessions
> > > >>>>>>>>>>> will
> > > >>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
> > > >>>>>> further
> > > >>>>>>>>>>>> improvement.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > >>>>>>>> jinfeng1995@gmail.com>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
> > > >>>>>> progress
> > > >>>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
> > > >>> Shammon
> > > >>>> FY,
> > > >>>>>>>> ShengKai
> > > >>>>>>>>>>> Fang
> > > >>>>>>>>>>>>>> and I
> > > >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
> > > >>>>>> solution
> > > >>>>>>>> for
> > > >>>>>>>>>>> Flink
> > > >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
> > > >>>> have
> > > >>>>>>>> updated
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> final
> > > >>>>>>>>>>>>>>>> solution in FLIP.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
> > > >>>>>>>> configuration
> > > >>>>>>>>>>>> similar
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> TableDescriptor.
> > > >>>>>>>>>>>>>>>> 2.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
> > > >>>>>>>>>>> storeCatalog(String
> > > >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
> > > >>>>>> CatalogDescriptor
> > > >>>>>>>>>>>>>>> getCatalog(String)
> > > >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
> > > >>>> Catalog
> > > >>>>>>>> instance.
> > > >>>>>>>>>>>> This
> > > >>>>>>>>>>>>>> way,
> > > >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
> > > >>> saving
> > > >>>>> and
> > > >>>>>>>>>>> retrieving
> > > >>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>> configurations without having to initialize
> > > >>>>>> catalogs.
> > > >>>>>>>>>>>>>>>> 3.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> The default registerCatalog(String
> > > >>> catalogName,
> > > >>>>>>> Catalog
> > > >>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
> > > >>>>>>> deprecated.
> > > >>>>>>>>>>>>>>>> 4.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> A new function registerCatalog(String
> > > >>>> catalogName,
> > > >>>>>>>>>>>>> CatalogDescriptor
> > > >>>>>>>>>>>>>>>> catalog) will be added to serve as the
> > > >>> default
> > > >>>>>>>> registration
> > > >>>>>>>>>>>>> function
> > > >>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> catalogs in CatalogManager.
> > > >>>>>>>>>>>>>>>> 5.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
> > > >>>> will
> > > >>>>>>>> remain
> > > >>>>>>>>>>>>> unchanged
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> save initialized catalogs.This means that
> > > >>>> deletion
> > > >>>>>>>>>> operations
> > > >>>>>>>>>>>> from
> > > >>>>>>>>>>>>>> one
> > > >>>>>>>>>>>>>>>> session won't synchronize with other
> > > >>> sessions.
> > > >>>>>>>>>>>>>>>> 6.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> To support multi-session synchronization
> > > >>>> scenarios
> > > >>>>>> for
> > > >>>>>>>>>>> deletions
> > > >>>>>>>>>>>>>> later
> > > >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
> > > >>>>>>>>>>> configurable.There
> > > >>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>> three possible situations:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
> > > >>>>>>>> automatically
> > > >>>>>>>>>>> clear
> > > >>>>>>>>>>>>>>>> long-unused catalogs.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
> > > >>>>>> getCatalog
> > > >>>>>>>>>>> creates a
> > > >>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>> instance.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> This is the document for discussion:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> This is the final proposal document:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thank you very much for your attention and
> > > >>>>> suggestions
> > > >>>>>> on
> > > >>>>>>>> this
> > > >>>>>>>>>>>>> FLIP. A
> > > >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
> > > >>> on
> > > >>>>> the
> > > >>>>>>>> entire
> > > >>>>>>>>>>>> design
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> organizing offline discussions.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
> > > >>>> about
> > > >>>>>>> this
> > > >>>>>>>> FLIP
> > > >>>>>>>>>>>>> please
> > > >>>>>>>>>>>>>>> feel
> > > >>>>>>>>>>>>>>>> free to share.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > >>>>>>>>>>> <jing@ververica.com.invalid
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
> > > >>> to
> > > >>>> me.
> > > >>>>>> We
> > > >>>>>>>> could
> > > >>>>>>>>>>>> still
> > > >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
> > > >>>> common
> > > >>>>>>>> built-in
> > > >>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
> > > >>> every
> > > >>>>>>>>>> CatalogSotre
> > > >>>>>>>>>>>>>>>>> implementation has to take care of it.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > >>>>>>>>>> jinfeng1995@gmail.com
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hi Jing,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks for your reply.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> > > >>>>> suggest
> > > >>>>>>> you
> > > >>>>>>>>>>>>> completely
> > > >>>>>>>>>>>>>>> walk
> > > >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
> > > >>> the
> > > >>>>> same
> > > >>>>>>>> time, I
> > > >>>>>>>>>>>>> greatly
> > > >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
> > > >>>> thoroughly
> > > >>>>>>>> checked
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> entire
> > > >>>>>>>>>>>>>>>> FLIP
> > > >>>>>>>>>>>>>>>>>> and made corrections to these issues.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
> > > >>> FLIP
> > > >>>>>>> design,
> > > >>>>>>>>>>>>>>> CatalogManager
> > > >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
> > > >>> being
> > > >>>>>>>> configured.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
> > > >>> was
> > > >>>> not
> > > >>>>>>>> necessary
> > > >>>>>>>>>>>>> because
> > > >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
> > > >>>> catalogs
> > > >>>>>> to
> > > >>>>>>>> store
> > > >>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>> instances.
> > > >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
> > > >>>>>>> Therefore, I
> > > >>>>>>>>>>>> modified
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>> part
> > > >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
> > > >>> Catalog>
> > > >>>>>>> catalogs
> > > >>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>> CatalogManager.
> > > >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
> > > >>> serve
> > > >>>> as
> > > >>>>>> the
> > > >>>>>>>>>> default
> > > >>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
> > > >>>>>>>> functionality of
> > > >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
> > > >>>>>>>>>>>>>>>>>> The previous plan that kept
> > > >>>>>>>> Map<String,Catalog>catalogs has
> > > >>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>> moved
> > > >>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>> Rejected Alternatives.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > >>>>>>>>>>>>> <jing@ververica.com.invalid
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> There are still many places contain
> > > >>>>> inconsistent
> > > >>>>>>>> content,
> > > >>>>>>>>>>>> e.g.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
> > > >>> used.
> > > >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
> > > >>>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
> > > >>> not
> > > >>>>> tell
> > > >>>>>>>> what the
> > > >>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>> doing.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> > > >>>>> suggest
> > > >>>>>>> you
> > > >>>>>>>>>>>> completely
> > > >>>>>>>>>>>>>>> walk
> > > >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
> > > >>> issues.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
> > > >>> mean
> > > >>>>> that
> > > >>>>>>> you
> > > >>>>>>>> will
> > > >>>>>>>>>>>> build
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
> > > >>> is a
> > > >>>>> very
> > > >>>>>>>>>> different
> > > >>>>>>>>>>>>>> design
> > > >>>>>>>>>>>>>>>>>> concept
> > > >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
> > > >>> I am
> > > >>>>> not
> > > >>>>>>>>>> mistaken,
> > > >>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
> > > >>>> work
> > > >>>>>>>> without
> > > >>>>>>>>>>>> Optional
> > > >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
> > > >>>>> reason
> > > >>>>>>>> why I
> > > >>>>>>>>>>>>> mentioned
> > > >>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
> > > >>>>> Optional
> > > >>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>> correct.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > >>>>>>>>>>>>> jinfeng1995@gmail.com>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hi Jing
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
> > > >>>> FLIP
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
> > > >>>> review. I
> > > >>>>>>> have
> > > >>>>>>>>>>> already
> > > >>>>>>>>>>>>> made
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> relevant changes.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
> > > >>> the
> > > >>>>>>>>>> multi-instance
> > > >>>>>>>>>>>>>>>> scenarios, I
> > > >>>>>>>>>>>>>>>>>> am
> > > >>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
> > > >>> multiple
> > > >>>>>>>> scenarios
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> current
> > > >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
> > > >>>>> remove
> > > >>>>>>>>>>> 'Map<String,
> > > >>>>>>>>>>>>>>>> Catalog>'
> > > >>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
> > > >>>>>>>>>> InMemoryCatalogStore.
> > > >>>>>>>>>>>> This
> > > >>>>>>>>>>>>>>> way,
> > > >>>>>>>>>>>>>>>>>> users
> > > >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
> > > >>>>> achieve
> > > >>>>>>>>>>>>> multi-instance
> > > >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
> > > >>>> think?
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > >>>>>>>>>>>>>>> <jing@ververica.com.invalid
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
> > > >>> are
> > > >>>>>> still
> > > >>>>>>>> some
> > > >>>>>>>>>>> NIT
> > > >>>>>>>>>>>>>> issues
> > > >>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> FLIP
> > > >>>>>>>>>>>>>>>>>>>>> like:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
> > > >>>> has
> > > >>>>>> been
> > > >>>>>>>> used
> > > >>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
> > > >>> example.
> > > >>>> It
> > > >>>>>>>> should be
> > > >>>>>>>>>>>> fine
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> use
> > > >>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
> > > >>> after
> > > >>>> you
> > > >>>>>>>> submit
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> PR.
> > > >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
> > > >>>> somewhere
> > > >>>>> in
> > > >>>>>>> the
> > > >>>>>>>>>>>> rejected
> > > >>>>>>>>>>>>>>>> section
> > > >>>>>>>>>>>>>>>>>>> which
> > > >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
> > > >>> it
> > > >>>>>>>> consistent.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
> > > >>>>>>>> multi-instance
> > > >>>>>>>>>>>>>>> scenarios,
> > > >>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>> am
> > > >>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
> > > >>> behaviour.
> > > >>>> If
> > > >>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
> > > >>>> once
> > > >>>>>> the
> > > >>>>>>>>>> related
> > > >>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>> has
> > > >>>>>>>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
> > > >>> another
> > > >>>>>>>> instance.
> > > >>>>>>>>>> Did
> > > >>>>>>>>>>> I
> > > >>>>>>>>>>>>> miss
> > > >>>>>>>>>>>>>>>>>>> something?
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
> > > >>> Jin
> > > >>>> <
> > > >>>>>>>>>>>>>>> jinfeng1995@gmail.com>
> > > >>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
> > > >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> @Jing
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
> > > >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
> > > >>> name, I
> > > >>>>>> have
> > > >>>>>>>>>> updated
> > > >>>>>>>>>>> it
> > > >>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>> documentation.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
> > > >>> should
> > > >>>> be
> > > >>>>>>>>>> implemented
> > > >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
> > > >>>>>> alternative
> > > >>>>>>> 1
> > > >>>>>>>>>> turns
> > > >>>>>>>>>>>> out
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
> > > >>> this
> > > >>>>> part
> > > >>>>>>> of
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> content.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
> > > >>>> forward,
> > > >>>>>> the
> > > >>>>>>>>>> option 2
> > > >>>>>>>>>>>>>>> solution
> > > >>>>>>>>>>>>>>>>>> looks
> > > >>>>>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
> > > >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
> > > >>>>> offline,
> > > >>>>>>> we
> > > >>>>>>>>>> think
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> solution
> > > >>>>>>>>>>>>>>>>>> 2
> > > >>>>>>>>>>>>>>>>>>>>> might
> > > >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
> > > >>>>>>>> inconsistency
> > > >>>>>>>>>>>> issues.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
> > > >>>>> descriptions
> > > >>>>>> in
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> content.
> > > >>>>>>>>>>>>>>>> Would
> > > >>>>>>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>>>>> to clean them up?
> > > >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
> > > >>>> document
> > > >>>>>> and
> > > >>>>>>>>>>>> appreciate
> > > >>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>>>>> suggestions.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> @Shammon
> > > >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
> > > >>> in
> > > >>>>>>>> `Rejected
> > > >>>>>>>>>>>>>>> Alternatives`
> > > >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
> > > >>> `Rejected
> > > >>>>>>>>>> Alternatives`.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Best
> > > >>>>>>>>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
> > > >>> Shammon
> > > >>>>> FY
> > > >>>>>> <
> > > >>>>>>>>>>>>>>> zjureel@gmail.com>
> > > >>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
> > > >>>>>> `Proposed
> > > >>>>>>>>>>> Changes`,
> > > >>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>>> put
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
> > > >>>>>>>> Alternatives`? I
> > > >>>>>>>>>>> think
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>>>>> help
> > > >>>>>>>>>>>>>>>>>>>> us
> > > >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
> > > >>> Jing
> > > >>>> Ge
> > > >>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
> > > >>> am
> > > >>>>>> still
> > > >>>>>>>>>> confused
> > > >>>>>>>>>>>>> after
> > > >>>>>>>>>>>>>>>>>>> completely
> > > >>>>>>>>>>>>>>>>>>>>>>> reading
> > > >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
> > > >>>> questions:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
> > > >>>>> registerCatalog()
> > > >>>>>>> and
> > > >>>>>>>>>>>>>> addCatalog()
> > > >>>>>>>>>>>>>>>> have
> > > >>>>>>>>>>>>>>>>>> no
> > > >>>>>>>>>>>>>>>>>>>> big
> > > >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
> > > >>>> One
> > > >>>>> of
> > > >>>>>>>> them is
> > > >>>>>>>>>>>>>>> responsible
> > > >>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>> data
> > > >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
> > > >>>>>> persistCatalog()?
> > > >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
> > > >>>> Map<String,
> > > >>>>>>>> Catalog>
> > > >>>>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>> as a
> > > >>>>>>>>>>>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
> > > >>> data
> > > >>>>>>>> persistence.
> > > >>>>>>>>>> I
> > > >>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>>> suggest
> > > >>>>>>>>>>>>>>>>>>>>>> describing
> > > >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
> > > >>>>> clearly
> > > >>>>>> in
> > > >>>>>>>> the
> > > >>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>> Some
> > > >>>>>>>>>>>>>>>>> common
> > > >>>>>>>>>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
> > > >>>> i.e.
> > > >>>>>> data
> > > >>>>>>>> in
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> store
> > > >>>>>>>>>>>>>>>>>>>>>>> should
> > > >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
> > > >>>> instance
> > > >>>>>>>> should be
> > > >>>>>>>>>>>> found
> > > >>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> store
> > > >>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
> > > >>>>>> initialized
> > > >>>>>>>> or it
> > > >>>>>>>>>>> will
> > > >>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>> lazy
> > > >>>>>>>>>>>>>>>>>>>>>> initialized)
> > > >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
> > > >>>>>>> consistency
> > > >>>>>>>> will
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>>>>> taken
> > > >>>>>>>>>>>>>>>> care
> > > >>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>> while
> > > >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
> > > >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
> > > >>>>>> forward,
> > > >>>>>>>> the
> > > >>>>>>>>>>>> option
> > > >>>>>>>>>>>>> 2
> > > >>>>>>>>>>>>>>>>> solution
> > > >>>>>>>>>>>>>>>>>>>> looks
> > > >>>>>>>>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
> > > >>>>>> because,
> > > >>>>>>>> afaiu,
> > > >>>>>>>>>>>>> issues
> > > >>>>>>>>>>>>>>>>>> mentioned
> > > >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
> > > >>>>> solved
> > > >>>>>>>> yet. Do
> > > >>>>>>>>>>> you
> > > >>>>>>>>>>>>>> still
> > > >>>>>>>>>>>>>>>>> want
> > > >>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> propose
> > > >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
> > > >>>> suggestions
> > > >>>>>> for
> > > >>>>>>>> both
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>>> them?
> > > >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
> > > >>>> there
> > > >>>>>> are
> > > >>>>>>>> some
> > > >>>>>>>>>>>>>>> inconsistent
> > > >>>>>>>>>>>>>>>>>>>>> descriptions
> > > >>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
> > > >>>> clean
> > > >>>>>>> them
> > > >>>>>>>> up?
> > > >>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
> > > >>> Feng
> > > >>>>>> Jin <
> > > >>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
> > > >>> and
> > > >>>> I
> > > >>>>>>>> completely
> > > >>>>>>>>>>>> agree
> > > >>>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>>>> point
> > > >>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>> view.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
> > > >>>>>> complicated
> > > >>>>>>>> the
> > > >>>>>>>>>>> whole
> > > >>>>>>>>>>>>>> issue.
> > > >>>>>>>>>>>>>>>>> First
> > > >>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>> foremost,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
> > > >>>> persistence
> > > >>>>>> of
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> Catalog's
> > > >>>>>>>>>>>>>>>>>>>> Configuration.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
> > > >>>>>> persistence
> > > >>>>>>>> for
> > > >>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>> Configuration,
> > > >>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
> > > >>> to
> > > >>>> the
> > > >>>>>>>> Catalog
> > > >>>>>>>>>>>>>> interface.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
> > > >>> Catalog
> > > >>>>>>>> instance
> > > >>>>>>>>>> to a
> > > >>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>> String>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
> > > >>>>>>>> implementation
> > > >>>>>>>>>> will
> > > >>>>>>>>>>>>> throw
> > > >>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>> exception.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
> > > >>>>>>>>>>>>>>>>>>>>>>>>> /**
> > > >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
> > > >>> the
> > > >>>>>>>> properties
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>>>> object.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> *
> > > >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
> > > >>> the
> > > >>>>>>>> properties
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>>>> object
> > > >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
> > > >>>>>>>> UnsupportedOperationException if
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> implementing
> > > >>>>>>>>>>>>>>>>>>>>> class
> > > >>>>>>>>>>>>>>>>>>>>>>> does
> > > >>>>>>>>>>>>>>>>>>>>>>>>> not override
> > > >>>>>>>>>>>>>>>>>>>>>>>>> * the default
> > > >>> implementation
> > > >>>> of
> > > >>>>>> this
> > > >>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>> */
> > > >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
> > > >>>>>>>> toProperties()
> > > >>>>>>>>>> {
> > > >>>>>>>>>>>>>>>>>>>>>>>>> throw new
> > > >>>>>>>>>>>> UnsupportedOperationException("Please
> > > >>>>>>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>>>>>>>>>>>> toProperties
> > > >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
> > > >>>> follows:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
> > > >>> configured a
> > > >>>>>>>>>> CatalogStore,
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> toConfiguration()
> > > >>>>>>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
> > > >>> registering a
> > > >>>>>>> Catalog
> > > >>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > > >>>> catalogName,
> > > >>>>>>>> Catalog
> > > >>>>>>>>>>>>> catalog).
> > > >>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
> > > >>>> Map<String,
> > > >>>>>>>> String>
> > > >>>>>>>>>>>>> properties
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>> saved
> > > >>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
> > > >>>>>> instances
> > > >>>>>>>> do
> > > >>>>>>>>>> not
> > > >>>>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> method,
> > > >>>>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
> > > >>>>>>>> CatalogStore,
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> toConfiguration()
> > > >>>>>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
> > > >>>>>>> consistency
> > > >>>>>>>> with
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> original
> > > >>>>>>>>>>>>>>>>>>>> process.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
> > > >>>>>> Catalog>
> > > >>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
> > > >>>>>> conflicts
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
> > > >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
> > > >>>>> itself
> > > >>>>>>> and
> > > >>>>>>>>>>> provide
> > > >>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>> dedicated
> > > >>>>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
> > > >>> loading,
> > > >>>>> such
> > > >>>>>>> as
> > > >>>>>>>>>>>>>>>>> ConfigurationCatalog
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> public class
> > > >>> ConfigurationCatalog
> > > >>>>>>>> implements
> > > >>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>> {
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>> ConfigurationCatalog(Map<String,
> > > >>>>>>> String>
> > > >>>>>>>>>>>>> properties)
> > > >>>>>>>>>>>>>> {
> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
> > > >>> the
> > > >>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
> > > >>> 10:03 AM
> > > >>>>>> Shammon
> > > >>>>>>>> FY <
> > > >>>>>>>>>>>>>>>>>> zjureel@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
> > > >>> the
> > > >>>>>>>> functionality
> > > >>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>> `CatalogStore`,
> > > >>>>>>>>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
> > > >>>> multiple
> > > >>>>>>>>>>>>> `registerCatalog`
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> > > >>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
> > > >>>>>> implementation
> > > >>>>>>> in
> > > >>>>>>>>>> Trino
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> same,
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
> > > >>> only
> > > >>>>> has
> > > >>>>>>> one
> > > >>>>>>>>>>>>>>> `createCatalog`,
> > > >>>>>>>>>>>>>>>>>> which
> > > >>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>> save
> > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
> > > >>>>>> `CatalogStore`.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
> > > >>>>>> another
> > > >>>>>>>>>>>>>>> `registerCatalog`
> > > >>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
> > > >>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
> > > >>> `Map<String,
> > > >>>>>>> Catalog>
> > > >>>>>>>>>>>> catalogs`
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>> `CatalogStore
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
> > > >>>> mentioned
> > > >>>>>>>> before,
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> `Map<String,
> > > >>>>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
> > > >>>>>>>> `CatalogManager`.
> > > >>>>>>>>>> How
> > > >>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>> saving
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
> > > >>> catalogs`
> > > >>>>> and
> > > >>>>>>>>>>>> `CatalogStore
> > > >>>>>>>>>>>>>>>>>>> catalogStore`
> > > >>>>>>>>>>>>>>>>>>>>>>>> together
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
> > > >>> added
> > > >>>> in
> > > >>>>>>>>>>>>> `CatalogManager`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > > >>> 5:22 PM
> > > >>>>> Feng
> > > >>>>>>>> Jin <
> > > >>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
> > > >>> I
> > > >>>> am
> > > >>>>>> very
> > > >>>>>>>> sorry
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>> misunderstanding
> > > >>>>>>>>>>>>>>>>>>>>>>>>> caused
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
> > > >>>>> original
> > > >>>>>>>>>>> discussion.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> > > >>>>>> pre-discussion
> > > >>>>>>>> [1]
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>> FLIP
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
> > > >>> such a
> > > >>>>>>>> discussion
> > > >>>>>>>>>>>> before.
> > > >>>>>>>>>>>>>>>>> However,
> > > >>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> designing
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
> > > >>> that
> > > >>>>> the
> > > >>>>>>>> logic of
> > > >>>>>>>>>>>>>>>>> CatalogManager
> > > >>>>>>>>>>>>>>>>>>>> itself
> > > >>>>>>>>>>>>>>>>>>>>>>>> doesn't
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
> > > >>>> cannot
> > > >>>>>>> only
> > > >>>>>>>>>>> persist
> > > >>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>> instances
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
> > > >>>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> might
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
> > > >>>> Catalogs,
> > > >>>>>>>> because
> > > >>>>>>>>>> in
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> end
> > > >>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>> still
> > > >>>>>>>>>>>>>>>>>>>>> need
> > > >>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> save
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
> > > >>>>> corresponding
> > > >>>>>> to
> > > >>>>>>>> the
> > > >>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>> instances.
> > > >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
> > > >>>>>>> CatalogStore
> > > >>>>>>>>>>>> interface
> > > >>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
> > > >>> this
> > > >>>>>> part, I
> > > >>>>>>>> also
> > > >>>>>>>>>>> took
> > > >>>>>>>>>>>>>>>>> inspiration
> > > >>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>> Trino's
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> > > >>>> handling
> > > >>>>>>>> catalog
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>> name
> > > >>>>>>>>>>>>>>>>>>>>> looks a
> > > >>>>>>>>>>>>>>>>>>>>>>>>> little
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> > > >>>> suggestion. I
> > > >>>>>>> will
> > > >>>>>>>>>>> provide
> > > >>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>> detailed
> > > >>>>>>>>>>>>>>>>>>>>>> description
> > > >>>>>>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
> > > >>> part,
> > > >>>>> and
> > > >>>>>>> add
> > > >>>>>>>> it
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
> > > >>>>>>>> `addCatalog`
> > > >>>>>>>>>>> still
> > > >>>>>>>>>>>>>> looks
> > > >>>>>>>>>>>>>>>>>> confused
> > > >>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
> > > >>>>>> catalog
> > > >>>>>>> to
> > > >>>>>>>>>>>>>> CatalogStore,
> > > >>>>>>>>>>>>>>>> how
> > > >>>>>>>>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>>>>>>>>> renaming
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
> > > >>> very
> > > >>>>>>> personal
> > > >>>>>>>>>>>> opinion,
> > > >>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>> decide
> > > >>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> take
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
> > > >>>>> intuitive
> > > >>>>>>> to
> > > >>>>>>>> me.
> > > >>>>>>>>>> I
> > > >>>>>>>>>>>>> don't
> > > >>>>>>>>>>>>>>> see
> > > >>>>>>>>>>>>>>>>> any
> > > >>>>>>>>>>>>>>>>>>>>> problem
> > > >>>>>>>>>>>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
> > > >>>>> statement,
> > > >>>>>>>> which
> > > >>>>>>>>>>>> method
> > > >>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>> called?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> > > >>>>>> `addCatalog`?
> > > >>>>>>>> I'm
> > > >>>>>>>>>>>>> wondering
> > > >>>>>>>>>>>>>>>>> whether
> > > >>>>>>>>>>>>>>>>>>> user
> > > >>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>> add a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
> > > >>>>>> stement.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
> > > >>>> original
> > > >>>>>>>> design
> > > >>>>>>>>>> was
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>> add
> > > >>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>> directly
> > > >>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
> > > >>> would
> > > >>>>>>> disrupt
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> existing
> > > >>>>>>>>>>>>>>>>> logic.
> > > >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
> > > >>>>>>>> configuration to
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>> initialize
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
> > > >>> the
> > > >>>>> same
> > > >>>>>>> time
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
> > > >>> neccessary
> > > >>>> to
> > > >>>>>>>> provide a
> > > >>>>>>>>>>>>> default
> > > >>>>>>>>>>>>>>>>>>>> implmentation
> > > >>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> > > >>>>> `CatalogStoreFactory`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
> > > >>>>>> otherwise
> > > >>>>>>> we
> > > >>>>>>>>>> would
> > > >>>>>>>>>>>>> need
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> introduce
> > > >>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> additional
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
> > > >>>> configuration
> > > >>>>>> for
> > > >>>>>>>> lazy
> > > >>>>>>>>>>>>> loading.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> > > >>>>>>> registration
> > > >>>>>>>> for
> > > >>>>>>>>>>>>> catalog.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
> > > >>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
> > > >>>>>> interface
> > > >>>>>>>>>> because
> > > >>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
> > > >>>>>>>> initialization
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
> > > >>>>>>>> inaccurate. It
> > > >>>>>>>>>>>>> should
> > > >>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> lazy
> > > >>>>>>>>>>>>>>>>>>>>>>> registration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
> > > >>>>>>>> registration. I
> > > >>>>>>>>>>> have
> > > >>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>> updated
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> title
> > > >>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > > >>>> 4:27 PM
> > > >>>>>>>> Shammon
> > > >>>>>>>>>> FY <
> > > >>>>>>>>>>>>>>>>>>> zjureel@gmail.com
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
> > > >>>>>>>> `registerCatalog`
> > > >>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
> > > >>>>> method
> > > >>>>>>>> named
> > > >>>>>>>>>>>>>> `addCatalog`
> > > >>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
> > > >>> the
> > > >>>>>> memory
> > > >>>>>>>>>> catalog
> > > >>>>>>>>>>>> is a
> > > >>>>>>>>>>>>>>>>> `cache`,
> > > >>>>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
> > > >>>>>> exposed
> > > >>>>>>> to
> > > >>>>>>>>>>> users.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> > > >>>>>> pre-discussion
> > > >>>>>>>> [1]
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>> Please
> > > >>>>>>>>>>>>>>>>>>>>>>> correct
> > > >>>>>>>>>>>>>>>>>>>>>>>>> me
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> if
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
> > > >>>>> conclusion
> > > >>>>>>> of
> > > >>>>>>>> that
> > > >>>>>>>>>>>>>>> discussion
> > > >>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>> use
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
> > > >>>>>> interface
> > > >>>>>>>> and
> > > >>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> different
> > > >>>>>>>>>>>>>>>>>>>>>>>>> stores
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> such
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
> > > >>>> external
> > > >>>>>>>> system.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
> > > >>>>> between
> > > >>>>>>> the
> > > >>>>>>>>>>> current
> > > >>>>>>>>>>>>> FLIP
> > > >>>>>>>>>>>>>>>>> design
> > > >>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
> > > >>> of
> > > >>>>> the
> > > >>>>>>>>>> discussion
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> thread
> > > >>>>>>>>>>>>>>>>>> [1] ?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>
> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > > >>>>> 3:41 PM
> > > >>>>>>>> yuxia <
> > > >>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
> > > >>> driving
> > > >>>>> this
> > > >>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> > > >>>>> handling
> > > >>>>>>>> catalog
> > > >>>>>>>>>>> with
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>>> name
> > > >>>>>>>>>>>>>>>>>>>>>> looks a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> little
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
> > > >>>>> it'll
> > > >>>>>> be
> > > >>>>>>>>>> better
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> explain
> > > >>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> java
> > > >>>>>>>>>>>>>>>>>>>>>>>>> doc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
> > > >>> give a
> > > >>>>>> brief
> > > >>>>>>>>>> example
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
> > > >>> name
> > > >>>>>>>> `addCatalog`
> > > >>>>>>>>>>>> still
> > > >>>>>>>>>>>>>>> looks
> > > >>>>>>>>>>>>>>>>>>> confused
> > > >>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
> > > >>>> storing
> > > >>>>>>>> catalog to
> > > >>>>>>>>>>>>>>>> CatalogStore,
> > > >>>>>>>>>>>>>>>>>> how
> > > >>>>>>>>>>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
> > > >>> It's
> > > >>>>> very
> > > >>>>>>>> personal
> > > >>>>>>>>>>>>>> opinion,
> > > >>>>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>> decide
> > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> take
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
> > > >>>>>>> statement,
> > > >>>>>>>>>> which
> > > >>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>> called?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> > > >>>>>>>> `addCatalog`?
> > > >>>>>>>>>> I'm
> > > >>>>>>>>>>>>>>> wondering
> > > >>>>>>>>>>>>>>>>>>> whether
> > > >>>>>>>>>>>>>>>>>>>>> user
> > > >>>>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> add a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
> > > >>> SQL
> > > >>>>>>>> stement.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
> > > >>>> neccessary
> > > >>>>> to
> > > >>>>>>>>>> provide a
> > > >>>>>>>>>>>>>> default
> > > >>>>>>>>>>>>>>>>>>>>> implmentation
> > > >>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> > > >>>>>>> `CatalogStoreFactory`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> > > >>>>>>>> registration
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>> catalog.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
> > > >>> catalog
> > > >>>>>> with
> > > >>>>>>>> CREATE
> > > >>>>>>>>>>>>>> CATALOG,
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> > > >>>> used
> > > >>>>>> by
> > > >>>>>>>>>> default.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
> > > >>>>> registration
> > > >>>>>> is
> > > >>>>>>>> the
> > > >>>>>>>>>>>> default
> > > >>>>>>>>>>>>>>>>> behavior,
> > > >>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>>>>>>>>>>> any
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> way
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
> > > >>> to
> > > >>>>>>>> synchronous
> > > >>>>>>>>>>>>>>> registration
> > > >>>>>>>>>>>>>>>>> just
> > > >>>>>>>>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>>>>>>> before?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
> > > >>>>> `addCatalog`
> > > >>>>>>> and
> > > >>>>>>>>>>>>>>>> `registerCatalog`
> > > >>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
> > > >>>>>>>> registration,
> > > >>>>>>>>>> it
> > > >>>>>>>>>>>> may
> > > >>>>>>>>>>>>>> well
> > > >>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>> CREATE
> > > >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
> > > >>>> but
> > > >>>>>> then
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> following
> > > >>>>>>>>>>>>>>>>> CREATE
> > > >>>>>>>>>>>>>>>>>>>> TABLE
> > > >>>>>>>>>>>>>>>>>>>>>>>>> statement
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
> > > >>> fail
> > > >>>>> to
> > > >>>>>>>> open.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
> > > >>>> change
> > > >>>>>>> which
> > > >>>>>>>>>>> should
> > > >>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>> highlighted
> > > >>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
> > > >>> part.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
> > > >>>> asynchronous
> > > >>>>>>>>>>>> registration, I
> > > >>>>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> expect
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
> > > >>>> open
> > > >>>>>> or
> > > >>>>>>>>>> register
> > > >>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> background,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
> > > >>> comments,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
> > > >>>> will
> > > >>>>> be
> > > >>>>>>>>>>> initialized
> > > >>>>>>>>>>>>> if
> > > >>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>> has
> > > >>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
> > > >>>> initialization
> > > >>>>>>>> process
> > > >>>>>>>>>>> fails,
> > > >>>>>>>>>>>>>> these
> > > >>>>>>>>>>>>>>>>>>>> statements
> > > >>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
> > > >>> successfully."
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
> > > >>> lazy
> > > >>>>>>>>>> initialization
> > > >>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
> > > >>>>>>>>>> jinfeng1995@gmail.com
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
> > > >>>>>>> dev@flink.apache.org
> > > >>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
> > > >>> 03日
> > > >>>> 下午
> > > >>>>>>>> 3:27:45
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
> > > >>>> 295:
> > > >>>>>>>> Support
> > > >>>>>>>>>>>>>> persistence
> > > >>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
> > > >>>>> registration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
> > > >>>> all
> > > >>>>>> for
> > > >>>>>>>> your
> > > >>>>>>>>>>>>> interest
> > > >>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> DISCUSS.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
> > > >>> catalog
> > > >>>>>> with
> > > >>>>>>>> the
> > > >>>>>>>>>> same
> > > >>>>>>>>>>>>> name
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>> both
> > > >>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
> > > >>>> crucial
> > > >>>>>>>> point.
> > > >>>>>>>>>>> Based
> > > >>>>>>>>>>>> on
> > > >>>>>>>>>>>>>> my
> > > >>>>>>>>>>>>>>>>>> current
> > > >>>>>>>>>>>>>>>>>>>>>> personal
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
> > > >>>>>> Map<String,
> > > >>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>> serve
> > > >>>>>>>>>>>>>>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
> > > >>> catalogs
> > > >>>>> and
> > > >>>>>>>> have
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> highest
> > > >>>>>>>>>>>>>>>>>>> priority.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
> > > >>> methods
> > > >>>>> that
> > > >>>>>>> can
> > > >>>>>>>>>> affect
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> > > >>> registerCatalog(String
> > > >>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
> > > >>>>> catalog
> > > >>>>>>>> instance
> > > >>>>>>>>>>>> into
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> > > >>>> unregisterCatalog(String
> > > >>>>>>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> > > >>> the
> > > >>>>>> catalog
> > > >>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>> corresponding
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
> > > >>>>> Catalog>
> > > >>>>>>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
> > > >>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
> > > >>>> retrieves
> > > >>>>>> the
> > > >>>>>>>>>>>>> corresponding
> > > >>>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
> > > >>> Catalog>
> > > >>>>>>>> catalogs. If
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>> does
> > > >>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>> exist,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
> > > >>>> corresponding
> > > >>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
> > > >>> puts
> > > >>>>> the
> > > >>>>>>>>>>> initialized
> > > >>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>>>>>> into
> > > >>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> > > >>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
> > > >>> methods
> > > >>>>>> only
> > > >>>>>>>> modify
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
> > > >>>>>>> catalogName,
> > > >>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>> String>
> > > >>>>>>>>>>>>>>>>>>>>>>> properties)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
> > > >>>>>>> properties
> > > >>>>>>>> to
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> catalogStore
> > > >>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>> checks
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
> > > >>>>>> catalogName
> > > >>>>>>>> with
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>> name.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> > > >>> removeCatalog(String
> > > >>>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> > > >>> the
> > > >>>>>>> specified
> > > >>>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> specified
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
> > > >>>>>>> catalogStore.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
> > > >>>> possible
> > > >>>>>>>> conflict
> > > >>>>>>>>>>>>>> scenarios:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
> > > >>> corresponding
> > > >>>>>>>> catalogName
> > > >>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
> > > >>> in
> > > >>>> the
> > > >>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>> Catalog>,
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > > >>>>>>>> catalogName,
> > > >>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
> > > >>> directly
> > > >>>>> saved
> > > >>>>>>> to
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
> > > >>> corresponding
> > > >>>>>>>> catalogName
> > > >>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>> both
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
> > > >>>>>>> Map<String,
> > > >>>>>>>>>>>> Catalog>,
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
> > > >>>>> catalog)
> > > >>>>>>>> method
> > > >>>>>>>>>>> will
> > > >>>>>>>>>>>>>> fail.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
> > > >>> corresponding
> > > >>>>>>>> catalogName
> > > >>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
> > > >>> the
> > > >>>>>>>>>>>> addCatalog(String
> > > >>>>>>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
> > > >>>> method
> > > >>>>>> can
> > > >>>>>>>>>> directly
> > > >>>>>>>>>>>>> save
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> properties
> > > >>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
> > > >>>>>>>>>> getCatalog(String
> > > >>>>>>>>>>>>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
> > > >>>> for
> > > >>>>>>>>>>> initialization
> > > >>>>>>>>>>>>>>> because
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> corresponding
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
> > > >>> already
> > > >>>>>> exists
> > > >>>>>>>> in
> > > >>>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
> > > >>>>>>>>>>>>> unregisterCatalog(String
> > > >>>>>>>>>>>>>>>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
> > > >>>>>>>> corresponding to
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> original
> > > >>>>>>>>>>>>>>>>>>>>> catalogName
> > > >>>>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
> > > >>> confuse
> > > >>>>>> users
> > > >>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
> > > >>>>> catalog)`
> > > >>>>>>> in
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>>> `Map<String,
> > > >>>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>> `registerCatalog(String
> > > >>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>> String>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> properties)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
> > > >>>> lead
> > > >>>>>> to
> > > >>>>>>>>>>>> confusion. I
> > > >>>>>>>>>>>>>>>> suggest
> > > >>>>>>>>>>>>>>>>>>>>> changing
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
> > > >>> to
> > > >>>>>>>>>>> addCatalog(String
> > > >>>>>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
> > > >>> as
> > > >>>>>>>> previously
> > > >>>>>>>>>>>>> mentioned
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
> > > >>>>> `registerCatalog(String
> > > >>>>>>>>>>>>>>> catalogName,Catalog
> > > >>>>>>>>>>>>>>>>>>>> catalog,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
> > > >>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
> > > >>>> already
> > > >>>>> an
> > > >>>>>>>>>> instance,
> > > >>>>>>>>>>>>>> adding
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> "lazyInit"
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
> > > >>>>>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
> > > >>>> necessarily
> > > >>>>>>>> result in
> > > >>>>>>>>>>> lazy
> > > >>>>>>>>>>>>>>>>>>> initialization.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
> > > >>>> about
> > > >>>>>>>>>> encryption
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
> > > >>>>>>> necessary,
> > > >>>>>>>> but
> > > >>>>>>>>>>>>> perhaps
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> encryption
> > > >>>>>>>>>>>>>>>>>>>>>>> logic
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
> > > >>> in
> > > >>>>> the
> > > >>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>> when
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> data
> > > >>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>> actually
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
> > > >>> the
> > > >>>>>>>>>>> FileCatalogStore
> > > >>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>> encrypt
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> data
> > > >>>>>>>>>>>>>>>>>>>>>>>>> when
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
> > > >>>>> while
> > > >>>>>>> the
> > > >>>>>>>>>>>>>>>> MemoryCatalogStore
> > > >>>>>>>>>>>>>>>>>>> does
> > > >>>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>> require
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
> > > >>> the
> > > >>>>>>>>>>>>>> `MemoryCatalogStore`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
> > > >>> necessary
> > > >>>> to
> > > >>>>>>> have a
> > > >>>>>>>>>>>>>>>>> MemoryCatalogStore
> > > >>>>>>>>>>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> default
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
> > > >>>> saves
> > > >>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>> configurations
> > > >>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>> memory.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
> > > >>> to
> > > >>>>>>>> implement
> > > >>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>> loading
> > > >>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
> > > >>> introduce
> > > >>>>>>>> additional
> > > >>>>>>>>>>> cache.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
> > > >>>>>>>> registration
> > > >>>>>>>>>>>> failed?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
> > > >>> critical
> > > >>>>>>> concern.
> > > >>>>>>>>>> When
> > > >>>>>>>>>>>>>>> executing
> > > >>>>>>>>>>>>>>>>> DDL,
> > > >>>>>>>>>>>>>>>>>>>> DQL,
> > > >>>>>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>> DML
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
> > > >>>> reference a
> > > >>>>>>>> specific
> > > >>>>>>>>>>>>> Catalog,
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
> > > >>> if it
> > > >>>>> has
> > > >>>>>>> not
> > > >>>>>>>>>> been
> > > >>>>>>>>>>>>>>>> initialized
> > > >>>>>>>>>>>>>>>>>> yet.
> > > >>>>>>>>>>>>>>>>>>>> If
> > > >>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
> > > >>>>> fails,
> > > >>>>>>>> these
> > > >>>>>>>>>>>>>> statements
> > > >>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>> executed
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
> > > >>>>>> Catalog>
> > > >>>>>>>>>> catalogs
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
> > > >>>> have
> > > >>>>>> the
> > > >>>>>>>> same
> > > >>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>> name,
> > > >>>>>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>>>>>>>>>> correponding
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> > > >>>>> instances.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
> > > >>>> resolved
> > > >>>>> by
> > > >>>>>>>>>> referring
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> previous
> > > >>>>>>>>>>>>>>>>>>>>>>>> responses.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
> > > >>> the
> > > >>>>>>> Catalog
> > > >>>>>>>>>> that
> > > >>>>>>>>>>>> has
> > > >>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>>>> most
> > > >>>>>>>>>>>>>>>>>>>>> recently
> > > >>>>>>>>>>>>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
> > > >>>> for
> > > >>>>>>>> subsequent
> > > >>>>>>>>>>>> use.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
> > > >>>>>> 10:58 PM
> > > >>>>>>>>>> Xianxun
> > > >>>>>>>>>>>> Ye
> > > >>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
> > > >>> this
> > > >>>>>> Flip,
> > > >>>>>>>> I do
> > > >>>>>>>>>>>>> believe
> > > >>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>> Flip
> > > >>>>>>>>>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
> > > >>>> work
> > > >>>>>> for
> > > >>>>>>>> also
> > > >>>>>>>>>>>>> manages a
> > > >>>>>>>>>>>>>>> lot
> > > >>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>> catalogs,
> > > >>>>>>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
> > > >>>> becomes
> > > >>>>>>> slow
> > > >>>>>>>>>>> because
> > > >>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>> loading a
> > > >>>>>>>>>>>>>>>>>>>>> number
> > > >>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
> > > >>> catalogs in
> > > >>>>> the
> > > >>>>>>>> user's
> > > >>>>>>>>>>>> Flink
> > > >>>>>>>>>>>>>> SQL
> > > >>>>>>>>>>>>>>>>>> through
> > > >>>>>>>>>>>>>>>>>>>>>> regular
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
> > > >>>>> catalogs
> > > >>>>>>> to
> > > >>>>>>>>>>> improve
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> speed
> > > >>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>> task
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
> > > >>> flip-295, I
> > > >>>>>> have
> > > >>>>>>>> some
> > > >>>>>>>>>>>>>> questions:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
> > > >>>>> catalog
> > > >>>>>>> with
> > > >>>>>>>>>>> CREATE
> > > >>>>>>>>>>>>>>> CATALOG,
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> > > >>>> used
> > > >>>>>> by
> > > >>>>>>>>>> default.
> > > >>>>>>>>>>>>> What
> > > >>>>>>>>>>>>>> if
> > > >>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
> > > >>>>> implement
> > > >>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>> registration?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
> > > >>> same
> > > >>>>>>>> question
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>>>>> Shammon’s
> > > >>>>>>>>>>>>>>>>>> first
> > > >>>>>>>>>>>>>>>>>>>>>>> comment.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> > > >>>>> catalogs
> > > >>>>>>> and
> > > >>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>> catalogstore
> > > >>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
> > > >>> have
> > > >>>> the
> > > >>>>>>> same
> > > >>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>> name,
> > > >>>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>>>>>>>> correponding
> > > >>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> > > >>>>> instances.
> > > >>>>>>> For
> > > >>>>>>>>>>> example:
> > > >>>>>>>>>>>>>>>>> catalog1->
> > > >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
> > > >>>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
> > > >>> catalog1 ->
> > > >>>>>>>> hiveCatalog
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> catalogstore.
> > > >>>>>>>>>>>>>>>>>>>>>>> Would
> > > >>>>>>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
> > > >>> should
> > > >>>>>> define
> > > >>>>>>>>>>> `catalogs`
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> `catalogstore`
> > > >>>>>>>>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
> > > >>> 11:13，Hang
> > > >>>>>> Ruan
> > > >>>>>>> <
> > > >>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> 写道：
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Feng.
I reviewed this FLIP again and found some little places that may need to
optimize.
1. `CatalogDescriptor` may need a private constructor.
2. The method `initCatalog` in `CatalogManager` is not introduced.
Best,
Hang
Feng Jin <ji...@gmail.com> 于2023年6月6日周二 21:17写道：
> Hi Leonard,
>
> Thanks for your reply.
>
> > 1. a How to construct a CatalogDescriptor ?
>
> I think it would be helpful to add a method for constructing a
> CatalogDescriptor, as you mentioned in 1.c. I will update the documentation
> later.
>
> > 1.b How to visit the fields ? Could we use Configuration instead of
> Map<String, String> ?
>
> I believe that the use of Map<String, String> options is only intended for
> creating a catalog and not for accessing internal parameters.
>
> Since all of the relevant parameters for CREATE CATALOG are also stored in
> Map<String, String> options, my understanding is that using Map<String,
> String> options should suffice.
>
> Here is the implementation of execute CREATE CATALOG statement.
> ```java
> private TableResultInternal createCatalog(CreateCatalogOperation operation)
> {
> String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
> try {
> String catalogName = operation.getCatalogName();
> Map<String, String> properties = operation.getProperties();
>
> Catalog catalog =
> FactoryUtil.createCatalog(
> catalogName,
> properties,
> tableConfig,
> resourceManager.getUserClassLoader());
> catalogManager.registerCatalog(catalogName, catalog);
>
> return TableResultImpl.TABLE_RESULT_OK;
> } catch (CatalogException e) {
> throw new ValidationException(exMsg, e);
> }
> }
> ```
>
>
> > 2. Do we have plan to offer a default CatalogStore if user didn’t config
> this?
>
> Yes, the in-memory catalogStore will be used as the default CatalogStore
> even if the user has not configured one
>
>
> Best,
> Feng
>
>
> On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xb...@gmail.com> wrote:
>
> > Hi, Feng
> >
> > Sorry for reply late, but I’ve some comments about the FLIP
> >
> >
> > 1. The introduced Public class CatalogDescriptor seems missed some
> > necessary component
> > a) How to construct a CatalogDescriptor ?
> > b) How to visit the fields ? Could we use Configuration instead of
> > Map<String, String> ?
> > c) Could we offer a built-in factory method to build a
> CatalogDescriptor
> > like
> > public static CatalogDescriptor of(String catalogName, Configuration
> > configuration)
> >
> > 2. The FLIP said “By default, there are two built-in CatalogStores
> > available: the In-Memory CatalogStore and the File CatalogStore” ,
> > Do we have plan to offer a default CatalogStore if user didn’t config
> > this? IIUC, users can obtains the benefits from lazy catalog
> > initialization If
> > we have a default catalogstore even it is in-memory.
> >
> > Best,
> > Leonard
> >
> >
> >
> > > On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > Thank you everyone for your valuable input. If there are no further
> > > questions or concerns about the FLIP[1], I would like to start voting
> > > tomorrow (6/7).
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com> wrote:
> > >
> > >> Hi Samrat,
> > >>
> > >> Thanks for your advice.
> > >>
> > >>> 1. The createCatalog method does not mention any exceptions being
> > >> thrown.
> > >>
> > >> CreateCatalog will throw CatalogException like registerCatalog. As
> > >> CatalogException is a RuntimeException,
> > >> there is no explicit declaration of throwing Exceptions in
> > CatalogManager
> > >> and TableEnvironment.
> > >> To avoid misunderstandings, I have added the "throw CatalogException"
> > flag
> > >> to the createCatalog method definition of CatalogStore.
> > >>
> > >>> 2. Could you please provide an exhaustive list of the supported
> kinds?
> > >>
> > >> Sure, the documentation now includes the configuration of the
> built-in
> > >> CatalogStore as well as how to configure a custom CatalogStore.
> > >>
> > >>
> > >> Best,
> > >> Feng
> > >>
> > >>
> > >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com>
> > wrote:
> > >>
> > >>> Hi Feng,
> > >>>
> > >>> Thank you for providing the proposal. I believe this feature will be
> > >>> highly
> > >>> valuable.
> > >>>
> > >>> I have a couple of inquiries:
> > >>>
> > >>> 1. According to the documentation [1], the createCatalog method does
> > not
> > >>> mention any exceptions being thrown. However, I would like to confirm
> > if
> > >>> it
> > >>> is always true that there will be no failures in all scenarios.
> Please
> > let
> > >>> me know if there is any additional information I may have missed.
> > >>>
> > >>> 2. Regarding the registration process using the
> > `table.catalog-store.kind`
> > >>> configuration, could you please provide an exhaustive list of the
> > >>> supported
> > >>> kinds?
> > >>> It would be great to have a comprehensive understanding of the
> > options
> > >>> available.
> > >>>
> > >>> Bests,
> > >>> Samrat
> > >>>
> > >>> [1]
> > >>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >>>
> > >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com>
> wrote:
> > >>>
> > >>>> Hi Hang and Jark
> > >>>>
> > >>>> Thank you very much for your reply.
> > >>>>
> > >>>> @Jark
> > >>>>> 1. Could you move the CatalogStore registration API to the "Public
> > >>>> Interface" section?
> > >>>>
> > >>>> Indeed, it is more reasonable.
> > >>>>
> > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> > >>>>
> > >>>> Sure, the names that you provided are indeed more logical and
> easier
> > >>> to
> > >>>> understand.
> > >>>>
> > >>>>> 3. About the open/close method in CatalogStoreFactory.
> > >>>>
> > >>>> I also agree with the proposed requirement scenario and design. I
> have
> > >>>> already made modifications to the interface.
> > >>>>
> > >>>> @Hang
> > >>>>
> > >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> > close
> > >>>> the resource.
> > >>>>
> > >>>> I have already added the missing method.
> > >>>>
> > >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > >>>>
> > >>>> Sorry for the misspelling, I updated this part.
> > >>>>
> > >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
> > methods
> > >>>> to init or close its resource
> > >>>>
> > >>>> This is indeed a scenario that needs to be considered, and I have
> > added
> > >>>> relevant methods accordingly.
> > >>>> Additionally, I have included some relevant description information
> in
> > >>> the
> > >>>> documentation to help others understand
> > >>>> why the open/close methods need to be added.
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>> Feng
> > >>>>
> > >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> > >>>>
> > >>>>> Hi Jing,
> > >>>>>
> > >>>>> Thank you for the update.
> > >>>>>
> > >>>>> 1. Could you move the CatalogStore registration API to the "Public
> > >>>>> Interface" section?
> > >>>>> "Proposed Changes" is more like a place to describe the
> > implementation
> > >>>>> details.
> > >>>>>
> > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> > >>> Besides,
> > >>>>> the config key
> > >>>>> name should be hierarchical[1]. Therefore, it may be better to use:
> > >>>>> "table.catalog-store.kind"
> > >>>>> "table.catalog-store.file.path"
> > >>>>>
> > >>>>> 3. I think Hang's suggestions make sense.
> > >>>>>
> > >>>>> Others look good to me.
> > >>>>>
> > >>>>> Best,
> > >>>>> Jark
> > >>>>>
> > >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
> > >>> wrote:
> > >>>>>
> > >>>>>> Hi, Feng.
> > >>>>>>
> > >>>>>> Thanks for the update.
> > >>>>>> The current design looks good to me. I have some minor comments.
> > >>>>>>
> > >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> > >>> close
> > >>>>> the
> > >>>>>> resource. For example, when storing the information in MySQL, the
> > >>> store
> > >>>>>> needs to open and close the connections.
> > >>>>>>
> > >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > >>>>>>
> > >>>>>> 3. About the usage in the sql gateway.
> > >>>>>> Considering the usage in sql gateway, the sql gateway may create a
> > >>>>>> CatalogStore for each session.
> > >>>>>> If we are using the MySqlCatalogStore, there would be so many
> > >>>>> connections.
> > >>>>>> How can we reuse the connection among these sessions?
> > >>>>>> I think sql gateway need to maintain a connection pool in
> > >>>>>> the CatalogStoreFactory and each session get its own connection
> from
> > >>>> the
> > >>>>>> pool when it is created.
> > >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close` methods
> > >>> to
> > >>>>> init
> > >>>>>> or close its resource.
> > >>>>>> Or is there a better way?
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Hang
> > >>>>>>
> > >>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> > >>>>>>
> > >>>>>>> Thanks Jingsong.
> > >>>>>>>
> > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
> > >>>>>>>
> > >>>>>>> +1 For this, I have already updated FLIP.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Feng
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> > >>> jingsonglee0@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Thanks Feng,
> > >>>>>>>>
> > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
> > >>>>>>>> functions are converted to createxxx from registerxxx.
> > >>>>>>>>
> > >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1995@gmail.com
> > >>>>
> > >>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Hi jark, thanks for your suggestion.
> > >>>>>>>>>
> > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> > >>> the
> > >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
> > >>>>> Otherwise,
> > >>>>>>>> there
> > >>>>>>>>> might be some data inconsistencies when CatalogStore is
> > >>> changed.
> > >>>>>>>>>
> > >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
> > >>>>>> TableEnv.
> > >>>>>>>>> Therefore, my current proposal is to add a method to configure
> > >>>> the
> > >>>>>>>>> CatalogStore in EnvironmentSettings.
> > >>>>>>>>>
> > >>>>>>>>> // Initialize a catalog Store
> > >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
> > >>>>>>>>>
> > >>>>>>>>> // set up the Table API
> > >>>>>>>>> final EnvironmentSettings settings =
> > >>>>>>>>>
> > >>> EnvironmentSettings.newInstance().inBatchMode()
> > >>>>>>>>> .withCatalogStore(catalogStore)
> > >>>>>>>>> .build();
> > >>>>>>>>>
> > >>>>>>>>> final TableEnvironment tableEnv =
> > >>>>>>>> TableEnvironment.create(settings);
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> > >>>> default
> > >>>>>>>> method,
> > >>>>>>>>> not an interface method?
> > >>>>>>>>>
> > >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
> > >>> that
> > >>>> as
> > >>>>> a
> > >>>>>>> new
> > >>>>>>>>> interface, the CatalogStoreFactory should not have a default
> > >>>> method
> > >>>>>> but
> > >>>>>>>> an
> > >>>>>>>>> interface method. I have already modified the interface.
> > >>>>>>>>>
> > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> > >>>>>>>>> deprecated`registerCatalog`.
> > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> > >>> mention
> > >>>> the
> > >>>>>>>> changed
> > >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> > >>>> other
> > >>>>>>>>> persisted catalog store) is used.
> > >>>>>>>>>
> > >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> [1].
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Feng
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
> > >>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Feng,
> > >>>>>>>>>>
> > >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
> > >>>>>>>>>> The current design looks pretty good to me. I just have some
> > >>>>> minor
> > >>>>>>>>>> comments.
> > >>>>>>>>>>
> > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> > >>> the
> > >>>>>>>> CatalogStore
> > >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
> > >>> there
> > >>>>>> might
> > >>>>>>> be
> > >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
> > >>>>>>>>>>
> > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> > >>>> default
> > >>>>>>>> method,
> > >>>>>>>>>> not an interface method?
> > >>>>>>>>>>
> > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> > >>>>> deprecated
> > >>>>>>>>>> `registerCatalog`.
> > >>>>>>>>>>
> > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> > >>> mention
> > >>>> the
> > >>>>>>>> changed
> > >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> > >>>>> other
> > >>>>>>>>>> persisted catalog store) is used.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Best,
> > >>>>>>>>>> Jark
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
> > >>> jinfeng1995@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
> > >>>>>>>>>>>
> > >>>>>>>>>>> @Ron
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
> > >>>> method,
> > >>>>>> do
> > >>>>>>> we
> > >>>>>>>>>> need
> > >>>>>>>>>>> to provide a default implementation?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Yes, we will provide a default
> > >>> InMemoryCatalogStoreFactory to
> > >>>>>>> create
> > >>>>>>>> an
> > >>>>>>>>>>> InMemoryCatalogStore.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
> > >>> initializing
> > >>>>> it,
> > >>>>>>>> whether
> > >>>>>>>>>> we
> > >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Yes, in the current design, catalogs are stored as
> > >>> snapshots,
> > >>>>> and
> > >>>>>>>> once
> > >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
> > >>>>>> Catalog>
> > >>>>>>>>>>> catalogs.
> > >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
> > >>> the
> > >>>>>> primary
> > >>>>>>>>>> source
> > >>>>>>>>>>> for obtaining the corresponding Catalog.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
> > >>>>>>>>>>> `catalog.store.path`?
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
> > >>> seem a
> > >>>>> bit
> > >>>>>>>>>> strange. I
> > >>>>>>>>>>> will update the FLIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> @Shammon
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thank you for the review. I have made the necessary
> > >>>>> corrections.
> > >>>>>>>>>>> Regarding the modifications made to the Public Interface,
> > >>> I
> > >>>>> have
> > >>>>>>> also
> > >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> Feng
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> > >>>> zjureel@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi feng,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for updating, I have some minor comments
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
> > >>>>>> `Public
> > >>>>>>>>>>>> Interfaces`, it is not a public interface.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
> > >>> and
> > >>>>>>>>>>>> `CatalogStoreFactory`
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
> > >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
> > >>>>> should
> > >>>>>> be
> > >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
> > >>>>>>>>>>>> catalogStore.get(catalogName);`
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Shammon FY
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
> > >>>> ron9.liu@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Feng
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
> > >>>> useful
> > >>>>>> for
> > >>>>>>>>>> catalog
> > >>>>>>>>>>>>> management.
> > >>>>>>>>>>>>> I have some small questions:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 1. Regarding the
> > >>> CatalogStoreFactory#createCatalogStore
> > >>>>>> method,
> > >>>>>>>> do we
> > >>>>>>>>>>>> need
> > >>>>>>>>>>>>> to provide a default implementation?
> > >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
> > >>>> initializing
> > >>>>>> it,
> > >>>>>>>>>> whether
> > >>>>>>>>>>> we
> > >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
> > >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
> > >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
> > >>> them to
> > >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>> Ron
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
> > >>> 21:19写道：
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi yuxia
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
> > >>>>> register
> > >>>>>>> the
> > >>>>>>>>>>> Catalog,
> > >>>>>>>>>>>>> we
> > >>>>>>>>>>>>>> initialize it and open it. right?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
> > >>>>>> original
> > >>>>>>>>>> CREATE
> > >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
> > >>>>>>>>>> registerCatalog
> > >>>>>>>>>>> so
> > >>>>>>>>>>>>>> that parameter validation can be performed.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
> > >>>>>>> reflected
> > >>>>>>>> in
> > >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
> > >>>>> catalog
> > >>>>>>>>>>>>> configurations,
> > >>>>>>>>>>>>>> only initialization is required in getCatalog.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
> > >>>>>>>> luoyuxia@alumni.sjtu.edu.cn
> > >>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi, Feng.
> > >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
> > >>>>>>>> initialization*.
> > >>>>>>>>>>> If
> > >>>>>>>>>>>>> i'm
> > >>>>>>>>>>>>>>> wrong, please correct me.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
> > >>>> access
> > >>>>>> the
> > >>>>>>>>>>> catalog,
> > >>>>>>>>>>>>> then
> > >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
> > >>>>> Changes,
> > >>>>>>>> once we
> > >>>>>>>>>>>>>> register
> > >>>>>>>>>>>>>>> the Catalog,
> > >>>>>>>>>>>>>>> we initialize it and open it. right?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>> Yuxia
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> ----- 原始邮件 -----
> > >>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > >>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
> > >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > >>>>>> Catalog
> > >>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>> and asynchronous registration
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> One of the major changes is that current design
> > >>> will
> > >>>>>>> provide
> > >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
> > >>>> of a
> > >>>>>>>> cache,
> > >>>>>>>>>>> which
> > >>>>>>>>>>>>>> means
> > >>>>>>>>>>>>>>> once it has been initialized, any changes done by
> > >>>> other
> > >>>>>>>> sessions
> > >>>>>>>>>>> will
> > >>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
> > >>>>>> further
> > >>>>>>>>>>>> improvement.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > >>>>>>>> jinfeng1995@gmail.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
> > >>>>>> progress
> > >>>>>>>> of
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
> > >>> Shammon
> > >>>> FY,
> > >>>>>>>> ShengKai
> > >>>>>>>>>>> Fang
> > >>>>>>>>>>>>>> and I
> > >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
> > >>>>>> solution
> > >>>>>>>> for
> > >>>>>>>>>>> Flink
> > >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
> > >>>> have
> > >>>>>>>> updated
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> final
> > >>>>>>>>>>>>>>>> solution in FLIP.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 1.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
> > >>>>>>>> configuration
> > >>>>>>>>>>>> similar
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> TableDescriptor.
> > >>>>>>>>>>>>>>>> 2.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
> > >>>>>>>>>>> storeCatalog(String
> > >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
> > >>>>>> CatalogDescriptor
> > >>>>>>>>>>>>>>> getCatalog(String)
> > >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
> > >>>> Catalog
> > >>>>>>>> instance.
> > >>>>>>>>>>>> This
> > >>>>>>>>>>>>>> way,
> > >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
> > >>> saving
> > >>>>> and
> > >>>>>>>>>>> retrieving
> > >>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>> configurations without having to initialize
> > >>>>>> catalogs.
> > >>>>>>>>>>>>>>>> 3.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The default registerCatalog(String
> > >>> catalogName,
> > >>>>>>> Catalog
> > >>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
> > >>>>>>> deprecated.
> > >>>>>>>>>>>>>>>> 4.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> A new function registerCatalog(String
> > >>>> catalogName,
> > >>>>>>>>>>>>> CatalogDescriptor
> > >>>>>>>>>>>>>>>> catalog) will be added to serve as the
> > >>> default
> > >>>>>>>> registration
> > >>>>>>>>>>>>> function
> > >>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> catalogs in CatalogManager.
> > >>>>>>>>>>>>>>>> 5.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
> > >>>> will
> > >>>>>>>> remain
> > >>>>>>>>>>>>> unchanged
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> save initialized catalogs.This means that
> > >>>> deletion
> > >>>>>>>>>> operations
> > >>>>>>>>>>>> from
> > >>>>>>>>>>>>>> one
> > >>>>>>>>>>>>>>>> session won't synchronize with other
> > >>> sessions.
> > >>>>>>>>>>>>>>>> 6.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> To support multi-session synchronization
> > >>>> scenarios
> > >>>>>> for
> > >>>>>>>>>>> deletions
> > >>>>>>>>>>>>>> later
> > >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
> > >>>>>>>>>>> configurable.There
> > >>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>> three possible situations:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
> > >>>>>>>> automatically
> > >>>>>>>>>>> clear
> > >>>>>>>>>>>>>>>> long-unused catalogs.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
> > >>>>>> getCatalog
> > >>>>>>>>>>> creates a
> > >>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>> instance.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This is the document for discussion:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This is the final proposal document:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thank you very much for your attention and
> > >>>>> suggestions
> > >>>>>> on
> > >>>>>>>> this
> > >>>>>>>>>>>>> FLIP. A
> > >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
> > >>> on
> > >>>>> the
> > >>>>>>>> entire
> > >>>>>>>>>>>> design
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> organizing offline discussions.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
> > >>>> about
> > >>>>>>> this
> > >>>>>>>> FLIP
> > >>>>>>>>>>>>> please
> > >>>>>>>>>>>>>>> feel
> > >>>>>>>>>>>>>>>> free to share.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > >>>>>>>>>>> <jing@ververica.com.invalid
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
> > >>> to
> > >>>> me.
> > >>>>>> We
> > >>>>>>>> could
> > >>>>>>>>>>>> still
> > >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
> > >>>> common
> > >>>>>>>> built-in
> > >>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
> > >>> every
> > >>>>>>>>>> CatalogSotre
> > >>>>>>>>>>>>>>>>> implementation has to take care of it.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > >>>>>>>>>> jinfeng1995@gmail.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi Jing,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for your reply.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> > >>>>> suggest
> > >>>>>>> you
> > >>>>>>>>>>>>> completely
> > >>>>>>>>>>>>>>> walk
> > >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
> > >>> the
> > >>>>> same
> > >>>>>>>> time, I
> > >>>>>>>>>>>>> greatly
> > >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
> > >>>> thoroughly
> > >>>>>>>> checked
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> entire
> > >>>>>>>>>>>>>>>> FLIP
> > >>>>>>>>>>>>>>>>>> and made corrections to these issues.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
> > >>> FLIP
> > >>>>>>> design,
> > >>>>>>>>>>>>>>> CatalogManager
> > >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
> > >>> being
> > >>>>>>>> configured.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
> > >>> was
> > >>>> not
> > >>>>>>>> necessary
> > >>>>>>>>>>>>> because
> > >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
> > >>>> catalogs
> > >>>>>> to
> > >>>>>>>> store
> > >>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>> instances.
> > >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
> > >>>>>>> Therefore, I
> > >>>>>>>>>>>> modified
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> part
> > >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
> > >>> Catalog>
> > >>>>>>> catalogs
> > >>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>> CatalogManager.
> > >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
> > >>> serve
> > >>>> as
> > >>>>>> the
> > >>>>>>>>>> default
> > >>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
> > >>>>>>>> functionality of
> > >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
> > >>>>>>>>>>>>>>>>>> The previous plan that kept
> > >>>>>>>> Map<String,Catalog>catalogs has
> > >>>>>>>>>>>> been
> > >>>>>>>>>>>>>>> moved
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> Rejected Alternatives.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > >>>>>>>>>>>>> <jing@ververica.com.invalid
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> There are still many places contain
> > >>>>> inconsistent
> > >>>>>>>> content,
> > >>>>>>>>>>>> e.g.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
> > >>> used.
> > >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
> > >>>>>>>> registerCatalog(String
> > >>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
> > >>> not
> > >>>>> tell
> > >>>>>>>> what the
> > >>>>>>>>>>>>> method
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> doing.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> > >>>>> suggest
> > >>>>>>> you
> > >>>>>>>>>>>> completely
> > >>>>>>>>>>>>>>> walk
> > >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
> > >>> issues.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
> > >>> mean
> > >>>>> that
> > >>>>>>> you
> > >>>>>>>> will
> > >>>>>>>>>>>> build
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
> > >>> is a
> > >>>>> very
> > >>>>>>>>>> different
> > >>>>>>>>>>>>>> design
> > >>>>>>>>>>>>>>>>>> concept
> > >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
> > >>> I am
> > >>>>> not
> > >>>>>>>>>> mistaken,
> > >>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
> > >>>> work
> > >>>>>>>> without
> > >>>>>>>>>>>> Optional
> > >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
> > >>>>> reason
> > >>>>>>>> why I
> > >>>>>>>>>>>>> mentioned
> > >>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
> > >>>>> Optional
> > >>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>> correct.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > >>>>>>>>>>>>> jinfeng1995@gmail.com>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi Jing
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
> > >>>> FLIP
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
> > >>>> review. I
> > >>>>>>> have
> > >>>>>>>>>>> already
> > >>>>>>>>>>>>> made
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> relevant changes.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
> > >>> the
> > >>>>>>>>>> multi-instance
> > >>>>>>>>>>>>>>>> scenarios, I
> > >>>>>>>>>>>>>>>>>> am
> > >>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
> > >>> multiple
> > >>>>>>>> scenarios
> > >>>>>>>>>> with
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> current
> > >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
> > >>>>> remove
> > >>>>>>>>>>> 'Map<String,
> > >>>>>>>>>>>>>>>> Catalog>'
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
> > >>>>>>>>>> InMemoryCatalogStore.
> > >>>>>>>>>>>> This
> > >>>>>>>>>>>>>>> way,
> > >>>>>>>>>>>>>>>>>> users
> > >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
> > >>>>> achieve
> > >>>>>>>>>>>>> multi-instance
> > >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
> > >>>> think?
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > >>>>>>>>>>>>>>> <jing@ververica.com.invalid
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
> > >>> are
> > >>>>>> still
> > >>>>>>>> some
> > >>>>>>>>>>> NIT
> > >>>>>>>>>>>>>> issues
> > >>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> FLIP
> > >>>>>>>>>>>>>>>>>>>>> like:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
> > >>>> has
> > >>>>>> been
> > >>>>>>>> used
> > >>>>>>>>>> as
> > >>>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
> > >>> example.
> > >>>> It
> > >>>>>>>> should be
> > >>>>>>>>>>>> fine
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> use
> > >>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
> > >>> after
> > >>>> you
> > >>>>>>>> submit
> > >>>>>>>>>> the
> > >>>>>>>>>>>> PR.
> > >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
> > >>>> somewhere
> > >>>>> in
> > >>>>>>> the
> > >>>>>>>>>>>> rejected
> > >>>>>>>>>>>>>>>> section
> > >>>>>>>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
> > >>> it
> > >>>>>>>> consistent.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
> > >>>>>>>> multi-instance
> > >>>>>>>>>>>>>>> scenarios,
> > >>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>> am
> > >>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
> > >>> behaviour.
> > >>>> If
> > >>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>> catalogs
> > >>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
> > >>>> once
> > >>>>>> the
> > >>>>>>>>>> related
> > >>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
> > >>> another
> > >>>>>>>> instance.
> > >>>>>>>>>> Did
> > >>>>>>>>>>> I
> > >>>>>>>>>>>>> miss
> > >>>>>>>>>>>>>>>>>>> something?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
> > >>> Jin
> > >>>> <
> > >>>>>>>>>>>>>>> jinfeng1995@gmail.com>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
> > >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> @Jing
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
> > >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
> > >>> name, I
> > >>>>>> have
> > >>>>>>>>>> updated
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> documentation.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
> > >>> should
> > >>>> be
> > >>>>>>>>>> implemented
> > >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
> > >>>>>> alternative
> > >>>>>>> 1
> > >>>>>>>>>> turns
> > >>>>>>>>>>>> out
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
> > >>> this
> > >>>>> part
> > >>>>>>> of
> > >>>>>>>> the
> > >>>>>>>>>>>>> content.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
> > >>>> forward,
> > >>>>>> the
> > >>>>>>>>>> option 2
> > >>>>>>>>>>>>>>> solution
> > >>>>>>>>>>>>>>>>>> looks
> > >>>>>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
> > >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
> > >>>>> offline,
> > >>>>>>> we
> > >>>>>>>>>> think
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> solution
> > >>>>>>>>>>>>>>>>>> 2
> > >>>>>>>>>>>>>>>>>>>>> might
> > >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
> > >>>>>>>> inconsistency
> > >>>>>>>>>>>> issues.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
> > >>>>> descriptions
> > >>>>>> in
> > >>>>>>>> the
> > >>>>>>>>>>>>> content.
> > >>>>>>>>>>>>>>>> Would
> > >>>>>>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>>>>> to clean them up?
> > >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
> > >>>> document
> > >>>>>> and
> > >>>>>>>>>>>> appreciate
> > >>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>>> suggestions.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> @Shammon
> > >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
> > >>> in
> > >>>>>>>> `Rejected
> > >>>>>>>>>>>>>>> Alternatives`
> > >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
> > >>> `Rejected
> > >>>>>>>>>> Alternatives`.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best
> > >>>>>>>>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
> > >>> Shammon
> > >>>>> FY
> > >>>>>> <
> > >>>>>>>>>>>>>>> zjureel@gmail.com>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
> > >>>>>> `Proposed
> > >>>>>>>>>>> Changes`,
> > >>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>> put
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
> > >>>>>>>> Alternatives`? I
> > >>>>>>>>>>> think
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>> help
> > >>>>>>>>>>>>>>>>>>>> us
> > >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
> > >>> Jing
> > >>>> Ge
> > >>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
> > >>> am
> > >>>>>> still
> > >>>>>>>>>> confused
> > >>>>>>>>>>>>> after
> > >>>>>>>>>>>>>>>>>>> completely
> > >>>>>>>>>>>>>>>>>>>>>>> reading
> > >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
> > >>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
> > >>>>> registerCatalog()
> > >>>>>>> and
> > >>>>>>>>>>>>>> addCatalog()
> > >>>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>>>>> no
> > >>>>>>>>>>>>>>>>>>>> big
> > >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
> > >>>> One
> > >>>>> of
> > >>>>>>>> them is
> > >>>>>>>>>>>>>>> responsible
> > >>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>> data
> > >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
> > >>>>>> persistCatalog()?
> > >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
> > >>>> Map<String,
> > >>>>>>>> Catalog>
> > >>>>>>>>>>>>> catalogs
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>> as a
> > >>>>>>>>>>>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
> > >>> data
> > >>>>>>>> persistence.
> > >>>>>>>>>> I
> > >>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>> suggest
> > >>>>>>>>>>>>>>>>>>>>>> describing
> > >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
> > >>>>> clearly
> > >>>>>> in
> > >>>>>>>> the
> > >>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>> Some
> > >>>>>>>>>>>>>>>>> common
> > >>>>>>>>>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
> > >>>> i.e.
> > >>>>>> data
> > >>>>>>>> in
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> store
> > >>>>>>>>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
> > >>>> instance
> > >>>>>>>> should be
> > >>>>>>>>>>>> found
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> store
> > >>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
> > >>>>>> initialized
> > >>>>>>>> or it
> > >>>>>>>>>>> will
> > >>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>> lazy
> > >>>>>>>>>>>>>>>>>>>>>> initialized)
> > >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
> > >>>>>>> consistency
> > >>>>>>>> will
> > >>>>>>>>>>> be
> > >>>>>>>>>>>>>> taken
> > >>>>>>>>>>>>>>>> care
> > >>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>> while
> > >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
> > >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
> > >>>>>> forward,
> > >>>>>>>> the
> > >>>>>>>>>>>> option
> > >>>>>>>>>>>>> 2
> > >>>>>>>>>>>>>>>>> solution
> > >>>>>>>>>>>>>>>>>>>> looks
> > >>>>>>>>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
> > >>>>>> because,
> > >>>>>>>> afaiu,
> > >>>>>>>>>>>>> issues
> > >>>>>>>>>>>>>>>>>> mentioned
> > >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
> > >>>>> solved
> > >>>>>>>> yet. Do
> > >>>>>>>>>>> you
> > >>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>>>>> want
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> propose
> > >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
> > >>>> suggestions
> > >>>>>> for
> > >>>>>>>> both
> > >>>>>>>>>> of
> > >>>>>>>>>>>>> them?
> > >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
> > >>>> there
> > >>>>>> are
> > >>>>>>>> some
> > >>>>>>>>>>>>>>> inconsistent
> > >>>>>>>>>>>>>>>>>>>>> descriptions
> > >>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
> > >>>> clean
> > >>>>>>> them
> > >>>>>>>> up?
> > >>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
> > >>> Feng
> > >>>>>> Jin <
> > >>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
> > >>> and
> > >>>> I
> > >>>>>>>> completely
> > >>>>>>>>>>>> agree
> > >>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>> point
> > >>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>> view.
> > >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
> > >>>>>> complicated
> > >>>>>>>> the
> > >>>>>>>>>>> whole
> > >>>>>>>>>>>>>> issue.
> > >>>>>>>>>>>>>>>>> First
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>> foremost,
> > >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
> > >>>> persistence
> > >>>>>> of
> > >>>>>>>> the
> > >>>>>>>>>>>>> Catalog's
> > >>>>>>>>>>>>>>>>>>>> Configuration.
> > >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
> > >>>>>> persistence
> > >>>>>>>> for
> > >>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>> Configuration,
> > >>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
> > >>> to
> > >>>> the
> > >>>>>>>> Catalog
> > >>>>>>>>>>>>>> interface.
> > >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
> > >>> Catalog
> > >>>>>>>> instance
> > >>>>>>>>>> to a
> > >>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>> String>
> > >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
> > >>>>>>>> implementation
> > >>>>>>>>>> will
> > >>>>>>>>>>>>> throw
> > >>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>> exception.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
> > >>>>>>>>>>>>>>>>>>>>>>>>> /**
> > >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
> > >>> the
> > >>>>>>>> properties
> > >>>>>>>>>> of
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>>>> object.
> > >>>>>>>>>>>>>>>>>>>>>>>>> *
> > >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
> > >>> the
> > >>>>>>>> properties
> > >>>>>>>>>> of
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>>>> object
> > >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
> > >>>>>>>> UnsupportedOperationException if
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> implementing
> > >>>>>>>>>>>>>>>>>>>>> class
> > >>>>>>>>>>>>>>>>>>>>>>> does
> > >>>>>>>>>>>>>>>>>>>>>>>>> not override
> > >>>>>>>>>>>>>>>>>>>>>>>>> * the default
> > >>> implementation
> > >>>> of
> > >>>>>> this
> > >>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>> */
> > >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
> > >>>>>>>> toProperties()
> > >>>>>>>>>> {
> > >>>>>>>>>>>>>>>>>>>>>>>>> throw new
> > >>>>>>>>>>>> UnsupportedOperationException("Please
> > >>>>>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>>>>>>>>> toProperties
> > >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
> > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
> > >>>> follows:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
> > >>> configured a
> > >>>>>>>>>> CatalogStore,
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> toConfiguration()
> > >>>>>>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
> > >>> registering a
> > >>>>>>> Catalog
> > >>>>>>>>>>>> instance
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > >>>> catalogName,
> > >>>>>>>> Catalog
> > >>>>>>>>>>>>> catalog).
> > >>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
> > >>>> Map<String,
> > >>>>>>>> String>
> > >>>>>>>>>>>>> properties
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>> saved
> > >>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
> > >>>>>> instances
> > >>>>>>>> do
> > >>>>>>>>>> not
> > >>>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>> method,
> > >>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
> > >>>>>>>> CatalogStore,
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> toConfiguration()
> > >>>>>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
> > >>>>>>> consistency
> > >>>>>>>> with
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> original
> > >>>>>>>>>>>>>>>>>>>> process.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
> > >>>>>> Catalog>
> > >>>>>>>>>>> catalogs
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
> > >>>>>> conflicts
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
> > >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
> > >>>>> itself
> > >>>>>>> and
> > >>>>>>>>>>> provide
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>> dedicated
> > >>>>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
> > >>> loading,
> > >>>>> such
> > >>>>>>> as
> > >>>>>>>>>>>>>>>>> ConfigurationCatalog
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> public class
> > >>> ConfigurationCatalog
> > >>>>>>>> implements
> > >>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>> {
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>> ConfigurationCatalog(Map<String,
> > >>>>>>> String>
> > >>>>>>>>>>>>> properties)
> > >>>>>>>>>>>>>> {
> > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
> > >>> the
> > >>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
> > >>> 10:03 AM
> > >>>>>> Shammon
> > >>>>>>>> FY <
> > >>>>>>>>>>>>>>>>>> zjureel@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
> > >>> the
> > >>>>>>>> functionality
> > >>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>> `CatalogStore`,
> > >>>>>>>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
> > >>>> multiple
> > >>>>>>>>>>>>> `registerCatalog`
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> > >>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
> > >>>>>> implementation
> > >>>>>>> in
> > >>>>>>>>>> Trino
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> same,
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
> > >>> only
> > >>>>> has
> > >>>>>>> one
> > >>>>>>>>>>>>>>> `createCatalog`,
> > >>>>>>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>> save
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
> > >>>>>> `CatalogStore`.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
> > >>>>>> another
> > >>>>>>>>>>>>>>> `registerCatalog`
> > >>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
> > >>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
> > >>> `Map<String,
> > >>>>>>> Catalog>
> > >>>>>>>>>>>> catalogs`
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> `CatalogStore
> > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
> > >>>> mentioned
> > >>>>>>>> before,
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>> `Map<String,
> > >>>>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
> > >>>>>>>> `CatalogManager`.
> > >>>>>>>>>> How
> > >>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>> saving
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
> > >>> catalogs`
> > >>>>> and
> > >>>>>>>>>>>> `CatalogStore
> > >>>>>>>>>>>>>>>>>>> catalogStore`
> > >>>>>>>>>>>>>>>>>>>>>>>> together
> > >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
> > >>> added
> > >>>> in
> > >>>>>>>>>>>>> `CatalogManager`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > >>> 5:22 PM
> > >>>>> Feng
> > >>>>>>>> Jin <
> > >>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
> > >>> I
> > >>>> am
> > >>>>>> very
> > >>>>>>>> sorry
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> misunderstanding
> > >>>>>>>>>>>>>>>>>>>>>>>>> caused
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
> > >>>>> original
> > >>>>>>>>>>> discussion.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> > >>>>>> pre-discussion
> > >>>>>>>> [1]
> > >>>>>>>>>> for
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> FLIP
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
> > >>> such a
> > >>>>>>>> discussion
> > >>>>>>>>>>>> before.
> > >>>>>>>>>>>>>>>>> However,
> > >>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> designing
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
> > >>> that
> > >>>>> the
> > >>>>>>>> logic of
> > >>>>>>>>>>>>>>>>> CatalogManager
> > >>>>>>>>>>>>>>>>>>>> itself
> > >>>>>>>>>>>>>>>>>>>>>>>> doesn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
> > >>>> cannot
> > >>>>>>> only
> > >>>>>>>>>>> persist
> > >>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>> instances
> > >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
> > >>>>>>>> registerCatalog(String
> > >>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>>>>>>>> might
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
> > >>>> Catalogs,
> > >>>>>>>> because
> > >>>>>>>>>> in
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> end
> > >>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>>>>>>>>> need
> > >>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> save
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
> > >>>>> corresponding
> > >>>>>> to
> > >>>>>>>> the
> > >>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>> instances.
> > >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
> > >>>>>>>>>>>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
> > >>>>>>> CatalogStore
> > >>>>>>>>>>>> interface
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
> > >>> this
> > >>>>>> part, I
> > >>>>>>>> also
> > >>>>>>>>>>> took
> > >>>>>>>>>>>>>>>>> inspiration
> > >>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>> Trino's
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> > >>>> handling
> > >>>>>>>> catalog
> > >>>>>>>>>> with
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>>>>>>> looks a
> > >>>>>>>>>>>>>>>>>>>>>>>>> little
> > >>>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> > >>>> suggestion. I
> > >>>>>>> will
> > >>>>>>>>>>> provide
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>> detailed
> > >>>>>>>>>>>>>>>>>>>>>> description
> > >>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
> > >>> part,
> > >>>>> and
> > >>>>>>> add
> > >>>>>>>> it
> > >>>>>>>>>> to
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
> > >>>>>>>> `addCatalog`
> > >>>>>>>>>>> still
> > >>>>>>>>>>>>>> looks
> > >>>>>>>>>>>>>>>>>> confused
> > >>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
> > >>>>>> catalog
> > >>>>>>> to
> > >>>>>>>>>>>>>> CatalogStore,
> > >>>>>>>>>>>>>>>> how
> > >>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>> renaming
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
> > >>> very
> > >>>>>>> personal
> > >>>>>>>>>>>> opinion,
> > >>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>> decide
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> take
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
> > >>>>> intuitive
> > >>>>>>> to
> > >>>>>>>> me.
> > >>>>>>>>>> I
> > >>>>>>>>>>>>> don't
> > >>>>>>>>>>>>>>> see
> > >>>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>> problem
> > >>>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>> it.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
> > >>>>> statement,
> > >>>>>>>> which
> > >>>>>>>>>>>> method
> > >>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>> called?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> > >>>>>> `addCatalog`?
> > >>>>>>>> I'm
> > >>>>>>>>>>>>> wondering
> > >>>>>>>>>>>>>>>>> whether
> > >>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>> add a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
> > >>>>>> stement.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
> > >>>> original
> > >>>>>>>> design
> > >>>>>>>>>> was
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>> add
> > >>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>> directly
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
> > >>> would
> > >>>>>>> disrupt
> > >>>>>>>> the
> > >>>>>>>>>>>>> existing
> > >>>>>>>>>>>>>>>>> logic.
> > >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
> > >>>>>>>>>>>>>>>>>>>>>>>>>> think
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
> > >>>>>>>> configuration to
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>> initialize
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
> > >>> the
> > >>>>> same
> > >>>>>>> time
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
> > >>> neccessary
> > >>>> to
> > >>>>>>>> provide a
> > >>>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>>>>> implmentation
> > >>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> > >>>>> `CatalogStoreFactory`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
> > >>>>>> otherwise
> > >>>>>>> we
> > >>>>>>>>>> would
> > >>>>>>>>>>>>> need
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> introduce
> > >>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>> additional
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
> > >>>> configuration
> > >>>>>> for
> > >>>>>>>> lazy
> > >>>>>>>>>>>>> loading.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> > >>>>>>> registration
> > >>>>>>>> for
> > >>>>>>>>>>>>> catalog.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
> > >>>>>>> registerCatalog(String
> > >>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
> > >>>>>> interface
> > >>>>>>>>>> because
> > >>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
> > >>>>>>>> initialization
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
> > >>>>>>>> inaccurate. It
> > >>>>>>>>>>>>> should
> > >>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> lazy
> > >>>>>>>>>>>>>>>>>>>>>>> registration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
> > >>>>>>>> registration. I
> > >>>>>>>>>>> have
> > >>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> title
> > >>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > >>>> 4:27 PM
> > >>>>>>>> Shammon
> > >>>>>>>>>> FY <
> > >>>>>>>>>>>>>>>>>>> zjureel@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
> > >>>>>>>> `registerCatalog`
> > >>>>>>>>>>>>> method
> > >>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
> > >>>>> method
> > >>>>>>>> named
> > >>>>>>>>>>>>>> `addCatalog`
> > >>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
> > >>> the
> > >>>>>> memory
> > >>>>>>>>>> catalog
> > >>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>> `cache`,
> > >>>>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>>>>>> think
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
> > >>>>>> exposed
> > >>>>>>> to
> > >>>>>>>>>>> users.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> > >>>>>> pre-discussion
> > >>>>>>>> [1]
> > >>>>>>>>>> for
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>> Please
> > >>>>>>>>>>>>>>>>>>>>>>> correct
> > >>>>>>>>>>>>>>>>>>>>>>>>> me
> > >>>>>>>>>>>>>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
> > >>>>> conclusion
> > >>>>>>> of
> > >>>>>>>> that
> > >>>>>>>>>>>>>>> discussion
> > >>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> use
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
> > >>>>>> interface
> > >>>>>>>> and
> > >>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> different
> > >>>>>>>>>>>>>>>>>>>>>>>>> stores
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> such
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
> > >>>> external
> > >>>>>>>> system.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
> > >>>>> between
> > >>>>>>> the
> > >>>>>>>>>>> current
> > >>>>>>>>>>>>> FLIP
> > >>>>>>>>>>>>>>>>> design
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
> > >>> of
> > >>>>> the
> > >>>>>>>>>> discussion
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> thread
> > >>>>>>>>>>>>>>>>>> [1] ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > >>>>> 3:41 PM
> > >>>>>>>> yuxia <
> > >>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
> > >>> driving
> > >>>>> this
> > >>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> > >>>>> handling
> > >>>>>>>> catalog
> > >>>>>>>>>>> with
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>>>>>>>> looks a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> little
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
> > >>>>> it'll
> > >>>>>> be
> > >>>>>>>>>> better
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>>> explain
> > >>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> java
> > >>>>>>>>>>>>>>>>>>>>>>>>> doc
> > >>>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
> > >>> give a
> > >>>>>> brief
> > >>>>>>>>>> example
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
> > >>> name
> > >>>>>>>> `addCatalog`
> > >>>>>>>>>>>> still
> > >>>>>>>>>>>>>>> looks
> > >>>>>>>>>>>>>>>>>>> confused
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
> > >>>> storing
> > >>>>>>>> catalog to
> > >>>>>>>>>>>>>>>> CatalogStore,
> > >>>>>>>>>>>>>>>>>> how
> > >>>>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
> > >>> It's
> > >>>>> very
> > >>>>>>>> personal
> > >>>>>>>>>>>>>> opinion,
> > >>>>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>> decide
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> take
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
> > >>>>>>> statement,
> > >>>>>>>>>> which
> > >>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>> called?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> > >>>>>>>> `addCatalog`?
> > >>>>>>>>>> I'm
> > >>>>>>>>>>>>>>> wondering
> > >>>>>>>>>>>>>>>>>>> whether
> > >>>>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>> add a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
> > >>> SQL
> > >>>>>>>> stement.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
> > >>>> neccessary
> > >>>>> to
> > >>>>>>>>>> provide a
> > >>>>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>>>>>> implmentation
> > >>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> > >>>>>>> `CatalogStoreFactory`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> > >>>>>>>> registration
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>> catalog.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
> > >>> catalog
> > >>>>>> with
> > >>>>>>>> CREATE
> > >>>>>>>>>>>>>> CATALOG,
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> > >>>> used
> > >>>>>> by
> > >>>>>>>>>> default.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
> > >>>>> registration
> > >>>>>> is
> > >>>>>>>> the
> > >>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>> behavior,
> > >>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>>>>>>> way
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
> > >>> to
> > >>>>>>>> synchronous
> > >>>>>>>>>>>>>>> registration
> > >>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>>>>>>> before?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
> > >>>>> `addCatalog`
> > >>>>>>> and
> > >>>>>>>>>>>>>>>> `registerCatalog`
> > >>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
> > >>>>>>>> registration,
> > >>>>>>>>>> it
> > >>>>>>>>>>>> may
> > >>>>>>>>>>>>>> well
> > >>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>> CREATE
> > >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
> > >>>> but
> > >>>>>> then
> > >>>>>>>> the
> > >>>>>>>>>>>>> following
> > >>>>>>>>>>>>>>>>> CREATE
> > >>>>>>>>>>>>>>>>>>>> TABLE
> > >>>>>>>>>>>>>>>>>>>>>>>>> statement
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
> > >>> fail
> > >>>>> to
> > >>>>>>>> open.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
> > >>>> change
> > >>>>>>> which
> > >>>>>>>>>>> should
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>> highlighted
> > >>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
> > >>> part.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
> > >>>> asynchronous
> > >>>>>>>>>>>> registration, I
> > >>>>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> expect
> > >>>>>>>>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
> > >>>> open
> > >>>>>> or
> > >>>>>>>>>> register
> > >>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> background,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
> > >>> comments,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
> > >>>> will
> > >>>>> be
> > >>>>>>>>>>> initialized
> > >>>>>>>>>>>>> if
> > >>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
> > >>>> initialization
> > >>>>>>>> process
> > >>>>>>>>>>> fails,
> > >>>>>>>>>>>>>> these
> > >>>>>>>>>>>>>>>>>>>> statements
> > >>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
> > >>> successfully."
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
> > >>> lazy
> > >>>>>>>>>> initialization
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
> > >>>>>>>>>> jinfeng1995@gmail.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
> > >>>>>>> dev@flink.apache.org
> > >>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
> > >>> 03日
> > >>>> 下午
> > >>>>>>>> 3:27:45
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
> > >>>> 295:
> > >>>>>>>> Support
> > >>>>>>>>>>>>>> persistence
> > >>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
> > >>>>> registration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
> > >>>> all
> > >>>>>> for
> > >>>>>>>> your
> > >>>>>>>>>>>>> interest
> > >>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>> DISCUSS.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
> > >>> catalog
> > >>>>>> with
> > >>>>>>>> the
> > >>>>>>>>>> same
> > >>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>> both
> > >>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
> > >>>> crucial
> > >>>>>>>> point.
> > >>>>>>>>>>> Based
> > >>>>>>>>>>>> on
> > >>>>>>>>>>>>>> my
> > >>>>>>>>>>>>>>>>>> current
> > >>>>>>>>>>>>>>>>>>>>>> personal
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
> > >>>>>> Map<String,
> > >>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>> catalogs
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>> serve
> > >>>>>>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
> > >>> catalogs
> > >>>>> and
> > >>>>>>>> have
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> highest
> > >>>>>>>>>>>>>>>>>>> priority.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
> > >>> methods
> > >>>>> that
> > >>>>>>> can
> > >>>>>>>>>> affect
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> > >>> registerCatalog(String
> > >>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
> > >>>>> catalog
> > >>>>>>>> instance
> > >>>>>>>>>>>> into
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> > >>>> unregisterCatalog(String
> > >>>>>>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> > >>> the
> > >>>>>> catalog
> > >>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>> corresponding
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
> > >>>>> Catalog>
> > >>>>>>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
> > >>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
> > >>>> retrieves
> > >>>>>> the
> > >>>>>>>>>>>>> corresponding
> > >>>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
> > >>> Catalog>
> > >>>>>>>> catalogs. If
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>> does
> > >>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>> exist,
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
> > >>>> corresponding
> > >>>>>>>>>>> configuration
> > >>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
> > >>> puts
> > >>>>> the
> > >>>>>>>>>>> initialized
> > >>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>>>>>> into
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> > >>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
> > >>> methods
> > >>>>>> only
> > >>>>>>>> modify
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
> > >>>>>>> catalogName,
> > >>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>> String>
> > >>>>>>>>>>>>>>>>>>>>>>> properties)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
> > >>>>>>> properties
> > >>>>>>>> to
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> catalogStore
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> checks
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
> > >>>>>> catalogName
> > >>>>>>>> with
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>> name.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> > >>> removeCatalog(String
> > >>>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> > >>> the
> > >>>>>>> specified
> > >>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> specified
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
> > >>>>>>> catalogStore.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
> > >>>> possible
> > >>>>>>>> conflict
> > >>>>>>>>>>>>>> scenarios:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
> > >>> corresponding
> > >>>>>>>> catalogName
> > >>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
> > >>> in
> > >>>> the
> > >>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>> Catalog>,
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > >>>>>>>> catalogName,
> > >>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
> > >>> directly
> > >>>>> saved
> > >>>>>>> to
> > >>>>>>>> the
> > >>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
> > >>> corresponding
> > >>>>>>>> catalogName
> > >>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> both
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
> > >>>>>>> Map<String,
> > >>>>>>>>>>>> Catalog>,
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
> > >>>>> catalog)
> > >>>>>>>> method
> > >>>>>>>>>>> will
> > >>>>>>>>>>>>>> fail.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
> > >>> corresponding
> > >>>>>>>> catalogName
> > >>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
> > >>> the
> > >>>>>>>>>>>> addCatalog(String
> > >>>>>>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
> > >>>> method
> > >>>>>> can
> > >>>>>>>>>> directly
> > >>>>>>>>>>>>> save
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> properties
> > >>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
> > >>>>>>>>>> getCatalog(String
> > >>>>>>>>>>>>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
> > >>>> for
> > >>>>>>>>>>> initialization
> > >>>>>>>>>>>>>>> because
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> corresponding
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
> > >>> already
> > >>>>>> exists
> > >>>>>>>> in
> > >>>>>>>>>>>> catalogs
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
> > >>>>>>>>>>>>> unregisterCatalog(String
> > >>>>>>>>>>>>>>>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
> > >>>>>>>> corresponding to
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>> original
> > >>>>>>>>>>>>>>>>>>>>> catalogName
> > >>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
> > >>> confuse
> > >>>>>> users
> > >>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
> > >>>>> catalog)`
> > >>>>>>> in
> > >>>>>>>> the
> > >>>>>>>>>>>>>>> `Map<String,
> > >>>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>> `registerCatalog(String
> > >>>>>>>> catalogName,
> > >>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>> String>
> > >>>>>>>>>>>>>>>>>>>>>>>>> properties)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
> > >>>> lead
> > >>>>>> to
> > >>>>>>>>>>>> confusion. I
> > >>>>>>>>>>>>>>>> suggest
> > >>>>>>>>>>>>>>>>>>>>> changing
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
> > >>> to
> > >>>>>>>>>>> addCatalog(String
> > >>>>>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
> > >>> as
> > >>>>>>>> previously
> > >>>>>>>>>>>>> mentioned
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
> > >>>>> `registerCatalog(String
> > >>>>>>>>>>>>>>> catalogName,Catalog
> > >>>>>>>>>>>>>>>>>>>> catalog,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
> > >>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
> > >>>> already
> > >>>>> an
> > >>>>>>>>>> instance,
> > >>>>>>>>>>>>>> adding
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> "lazyInit"
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
> > >>>>>>>>>> registerCatalog(String
> > >>>>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
> > >>>> necessarily
> > >>>>>>>> result in
> > >>>>>>>>>>> lazy
> > >>>>>>>>>>>>>>>>>>> initialization.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
> > >>>> about
> > >>>>>>>>>> encryption
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
> > >>>>>>> necessary,
> > >>>>>>>> but
> > >>>>>>>>>>>>> perhaps
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> encryption
> > >>>>>>>>>>>>>>>>>>>>>>> logic
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
> > >>> in
> > >>>>> the
> > >>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>> when
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> data
> > >>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>> actually
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
> > >>> the
> > >>>>>>>>>>> FileCatalogStore
> > >>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>> encrypt
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> data
> > >>>>>>>>>>>>>>>>>>>>>>>>> when
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
> > >>>>> while
> > >>>>>>> the
> > >>>>>>>>>>>>>>>> MemoryCatalogStore
> > >>>>>>>>>>>>>>>>>>> does
> > >>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>> require
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
> > >>> the
> > >>>>>>>>>>>>>> `MemoryCatalogStore`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
> > >>> necessary
> > >>>> to
> > >>>>>>> have a
> > >>>>>>>>>>>>>>>>> MemoryCatalogStore
> > >>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
> > >>>> saves
> > >>>>>>>> Catalog
> > >>>>>>>>>>>>>>>> configurations
> > >>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> memory.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
> > >>> to
> > >>>>>>>> implement
> > >>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>> loading
> > >>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
> > >>> introduce
> > >>>>>>>> additional
> > >>>>>>>>>>> cache.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
> > >>>>>>>> registration
> > >>>>>>>>>>>> failed?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
> > >>> critical
> > >>>>>>> concern.
> > >>>>>>>>>> When
> > >>>>>>>>>>>>>>> executing
> > >>>>>>>>>>>>>>>>> DDL,
> > >>>>>>>>>>>>>>>>>>>> DQL,
> > >>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>> DML
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
> > >>>> reference a
> > >>>>>>>> specific
> > >>>>>>>>>>>>> Catalog,
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
> > >>> if it
> > >>>>> has
> > >>>>>>> not
> > >>>>>>>>>> been
> > >>>>>>>>>>>>>>>> initialized
> > >>>>>>>>>>>>>>>>>> yet.
> > >>>>>>>>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
> > >>>>> fails,
> > >>>>>>>> these
> > >>>>>>>>>>>>>> statements
> > >>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>> executed
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
> > >>>>>> Catalog>
> > >>>>>>>>>> catalogs
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
> > >>>> have
> > >>>>>> the
> > >>>>>>>> same
> > >>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>> name,
> > >>>>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>>>>>> correponding
> > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> > >>>>> instances.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
> > >>>> resolved
> > >>>>> by
> > >>>>>>>>>> referring
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> previous
> > >>>>>>>>>>>>>>>>>>>>>>>> responses.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
> > >>> the
> > >>>>>>> Catalog
> > >>>>>>>>>> that
> > >>>>>>>>>>>> has
> > >>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>> recently
> > >>>>>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
> > >>>> for
> > >>>>>>>> subsequent
> > >>>>>>>>>>>> use.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
> > >>>>>> 10:58 PM
> > >>>>>>>>>> Xianxun
> > >>>>>>>>>>>> Ye
> > >>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
> > >>> this
> > >>>>>> Flip,
> > >>>>>>>> I do
> > >>>>>>>>>>>>> believe
> > >>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>> Flip
> > >>>>>>>>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
> > >>>> work
> > >>>>>> for
> > >>>>>>>> also
> > >>>>>>>>>>>>> manages a
> > >>>>>>>>>>>>>>> lot
> > >>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>> catalogs,
> > >>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
> > >>>> becomes
> > >>>>>>> slow
> > >>>>>>>>>>> because
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>> loading a
> > >>>>>>>>>>>>>>>>>>>>> number
> > >>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
> > >>> catalogs in
> > >>>>> the
> > >>>>>>>> user's
> > >>>>>>>>>>>> Flink
> > >>>>>>>>>>>>>> SQL
> > >>>>>>>>>>>>>>>>>> through
> > >>>>>>>>>>>>>>>>>>>>>> regular
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
> > >>>>> catalogs
> > >>>>>>> to
> > >>>>>>>>>>> improve
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> speed
> > >>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>> task
> > >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
> > >>> flip-295, I
> > >>>>>> have
> > >>>>>>>> some
> > >>>>>>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
> > >>>>> catalog
> > >>>>>>> with
> > >>>>>>>>>>> CREATE
> > >>>>>>>>>>>>>>> CATALOG,
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> > >>>> used
> > >>>>>> by
> > >>>>>>>>>> default.
> > >>>>>>>>>>>>> What
> > >>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
> > >>>>> implement
> > >>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>> registration?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
> > >>> same
> > >>>>>>>> question
> > >>>>>>>>>> with
> > >>>>>>>>>>>>>>> Shammon’s
> > >>>>>>>>>>>>>>>>>> first
> > >>>>>>>>>>>>>>>>>>>>>>> comment.
> > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> > >>>>> catalogs
> > >>>>>>> and
> > >>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>> catalogstore
> > >>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
> > >>> have
> > >>>> the
> > >>>>>>> same
> > >>>>>>>>>>> catalog
> > >>>>>>>>>>>>>> name,
> > >>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>>>> correponding
> > >>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> > >>>>> instances.
> > >>>>>>> For
> > >>>>>>>>>>> example:
> > >>>>>>>>>>>>>>>>> catalog1->
> > >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
> > >>>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
> > >>> catalog1 ->
> > >>>>>>>> hiveCatalog
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> catalogstore.
> > >>>>>>>>>>>>>>>>>>>>>>> Would
> > >>>>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
> > >>> should
> > >>>>>> define
> > >>>>>>>>>>> `catalogs`
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> `catalogstore`
> > >>>>>>>>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
> > >>> 11:13，Hang
> > >>>>>> Ruan
> > >>>>>>> <
> > >>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> 写道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Leonard,
Thanks for your reply.
> 1. a How to construct a CatalogDescriptor ?
I think it would be helpful to add a method for constructing a
CatalogDescriptor, as you mentioned in 1.c. I will update the documentation
later.
> 1.b How to visit the fields ? Could we use Configuration instead of
Map<String, String> ?
I believe that the use of Map<String, String> options is only intended for
creating a catalog and not for accessing internal parameters.
Since all of the relevant parameters for CREATE CATALOG are also stored in
Map<String, String> options, my understanding is that using Map<String,
String> options should suffice.
Here is the implementation of execute CREATE CATALOG statement.
```java
private TableResultInternal createCatalog(CreateCatalogOperation operation)
{
String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
try {
String catalogName = operation.getCatalogName();
Map<String, String> properties = operation.getProperties();
Catalog catalog =
FactoryUtil.createCatalog(
catalogName,
properties,
tableConfig,
resourceManager.getUserClassLoader());
catalogManager.registerCatalog(catalogName, catalog);
return TableResultImpl.TABLE_RESULT_OK;
} catch (CatalogException e) {
throw new ValidationException(exMsg, e);
}
}
```
> 2. Do we have plan to offer a default CatalogStore if user didn’t config
this?
Yes, the in-memory catalogStore will be used as the default CatalogStore
even if the user has not configured one
Best,
Feng
On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xb...@gmail.com> wrote:
> Hi, Feng
>
> Sorry for reply late, but I’ve some comments about the FLIP
>
>
> 1. The introduced Public class CatalogDescriptor seems missed some
> necessary component
> a) How to construct a CatalogDescriptor ?
> b) How to visit the fields ? Could we use Configuration instead of
> Map<String, String> ?
> c) Could we offer a built-in factory method to build a CatalogDescriptor
> like
> public static CatalogDescriptor of(String catalogName, Configuration
> configuration)
>
> 2. The FLIP said “By default, there are two built-in CatalogStores
> available: the In-Memory CatalogStore and the File CatalogStore” ,
> Do we have plan to offer a default CatalogStore if user didn’t config
> this? IIUC, users can obtains the benefits from lazy catalog
> initialization If
> we have a default catalogstore even it is in-memory.
>
> Best,
> Leonard
>
>
>
> > On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > Thank you everyone for your valuable input. If there are no further
> > questions or concerns about the FLIP[1], I would like to start voting
> > tomorrow (6/7).
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >
> >
> > Best,
> > Feng
> >
> >
> > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com> wrote:
> >
> >> Hi Samrat,
> >>
> >> Thanks for your advice.
> >>
> >>> 1. The createCatalog method does not mention any exceptions being
> >> thrown.
> >>
> >> CreateCatalog will throw CatalogException like registerCatalog. As
> >> CatalogException is a RuntimeException,
> >> there is no explicit declaration of throwing Exceptions in
> CatalogManager
> >> and TableEnvironment.
> >> To avoid misunderstandings, I have added the "throw CatalogException"
> flag
> >> to the createCatalog method definition of CatalogStore.
> >>
> >>> 2. Could you please provide an exhaustive list of the supported kinds?
> >>
> >> Sure, the documentation now includes the configuration of the built-in
> >> CatalogStore as well as how to configure a custom CatalogStore.
> >>
> >>
> >> Best,
> >> Feng
> >>
> >>
> >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com>
> wrote:
> >>
> >>> Hi Feng,
> >>>
> >>> Thank you for providing the proposal. I believe this feature will be
> >>> highly
> >>> valuable.
> >>>
> >>> I have a couple of inquiries:
> >>>
> >>> 1. According to the documentation [1], the createCatalog method does
> not
> >>> mention any exceptions being thrown. However, I would like to confirm
> if
> >>> it
> >>> is always true that there will be no failures in all scenarios. Please
> let
> >>> me know if there is any additional information I may have missed.
> >>>
> >>> 2. Regarding the registration process using the
> `table.catalog-store.kind`
> >>> configuration, could you please provide an exhaustive list of the
> >>> supported
> >>> kinds?
> >>> It would be great to have a comprehensive understanding of the
> options
> >>> available.
> >>>
> >>> Bests,
> >>> Samrat
> >>>
> >>> [1]
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>
> >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
> >>>
> >>>> Hi Hang and Jark
> >>>>
> >>>> Thank you very much for your reply.
> >>>>
> >>>> @Jark
> >>>>> 1. Could you move the CatalogStore registration API to the "Public
> >>>> Interface" section?
> >>>>
> >>>> Indeed, it is more reasonable.
> >>>>
> >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> >>>>
> >>>> Sure, the names that you provided are indeed more logical and easier
> >>> to
> >>>> understand.
> >>>>
> >>>>> 3. About the open/close method in CatalogStoreFactory.
> >>>>
> >>>> I also agree with the proposed requirement scenario and design. I have
> >>>> already made modifications to the interface.
> >>>>
> >>>> @Hang
> >>>>
> >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> close
> >>>> the resource.
> >>>>
> >>>> I have already added the missing method.
> >>>>
> >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >>>>
> >>>> Sorry for the misspelling, I updated this part.
> >>>>
> >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
> methods
> >>>> to init or close its resource
> >>>>
> >>>> This is indeed a scenario that needs to be considered, and I have
> added
> >>>> relevant methods accordingly.
> >>>> Additionally, I have included some relevant description information in
> >>> the
> >>>> documentation to help others understand
> >>>> why the open/close methods need to be added.
> >>>>
> >>>>
> >>>> Best,
> >>>> Feng
> >>>>
> >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> >>>>
> >>>>> Hi Jing,
> >>>>>
> >>>>> Thank you for the update.
> >>>>>
> >>>>> 1. Could you move the CatalogStore registration API to the "Public
> >>>>> Interface" section?
> >>>>> "Proposed Changes" is more like a place to describe the
> implementation
> >>>>> details.
> >>>>>
> >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> >>> Besides,
> >>>>> the config key
> >>>>> name should be hierarchical[1]. Therefore, it may be better to use:
> >>>>> "table.catalog-store.kind"
> >>>>> "table.catalog-store.file.path"
> >>>>>
> >>>>> 3. I think Hang's suggestions make sense.
> >>>>>
> >>>>> Others look good to me.
> >>>>>
> >>>>> Best,
> >>>>> Jark
> >>>>>
> >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> Hi, Feng.
> >>>>>>
> >>>>>> Thanks for the update.
> >>>>>> The current design looks good to me. I have some minor comments.
> >>>>>>
> >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> >>> close
> >>>>> the
> >>>>>> resource. For example, when storing the information in MySQL, the
> >>> store
> >>>>>> needs to open and close the connections.
> >>>>>>
> >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >>>>>>
> >>>>>> 3. About the usage in the sql gateway.
> >>>>>> Considering the usage in sql gateway, the sql gateway may create a
> >>>>>> CatalogStore for each session.
> >>>>>> If we are using the MySqlCatalogStore, there would be so many
> >>>>> connections.
> >>>>>> How can we reuse the connection among these sessions?
> >>>>>> I think sql gateway need to maintain a connection pool in
> >>>>>> the CatalogStoreFactory and each session get its own connection from
> >>>> the
> >>>>>> pool when it is created.
> >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close` methods
> >>> to
> >>>>> init
> >>>>>> or close its resource.
> >>>>>> Or is there a better way?
> >>>>>>
> >>>>>> Best,
> >>>>>> Hang
> >>>>>>
> >>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> >>>>>>
> >>>>>>> Thanks Jingsong.
> >>>>>>>
> >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
> >>>>>>>
> >>>>>>> +1 For this, I have already updated FLIP.
> >>>>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Feng
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> >>> jingsonglee0@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks Feng,
> >>>>>>>>
> >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
> >>>>>>>> functions are converted to createxxx from registerxxx.
> >>>>>>>>
> >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1995@gmail.com
> >>>>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi jark, thanks for your suggestion.
> >>>>>>>>>
> >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> >>> the
> >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
> >>>>> Otherwise,
> >>>>>>>> there
> >>>>>>>>> might be some data inconsistencies when CatalogStore is
> >>> changed.
> >>>>>>>>>
> >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
> >>>>>> TableEnv.
> >>>>>>>>> Therefore, my current proposal is to add a method to configure
> >>>> the
> >>>>>>>>> CatalogStore in EnvironmentSettings.
> >>>>>>>>>
> >>>>>>>>> // Initialize a catalog Store
> >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
> >>>>>>>>>
> >>>>>>>>> // set up the Table API
> >>>>>>>>> final EnvironmentSettings settings =
> >>>>>>>>>
> >>> EnvironmentSettings.newInstance().inBatchMode()
> >>>>>>>>> .withCatalogStore(catalogStore)
> >>>>>>>>> .build();
> >>>>>>>>>
> >>>>>>>>> final TableEnvironment tableEnv =
> >>>>>>>> TableEnvironment.create(settings);
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> >>>> default
> >>>>>>>> method,
> >>>>>>>>> not an interface method?
> >>>>>>>>>
> >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
> >>> that
> >>>> as
> >>>>> a
> >>>>>>> new
> >>>>>>>>> interface, the CatalogStoreFactory should not have a default
> >>>> method
> >>>>>> but
> >>>>>>>> an
> >>>>>>>>> interface method. I have already modified the interface.
> >>>>>>>>>
> >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> >>>>>>>>> deprecated`registerCatalog`.
> >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> >>> mention
> >>>> the
> >>>>>>>> changed
> >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> >>>> other
> >>>>>>>>> persisted catalog store) is used.
> >>>>>>>>>
> >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [1].
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Feng
> >>>>>>>>>
> >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Feng,
> >>>>>>>>>>
> >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
> >>>>>>>>>> The current design looks pretty good to me. I just have some
> >>>>> minor
> >>>>>>>>>> comments.
> >>>>>>>>>>
> >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> >>> the
> >>>>>>>> CatalogStore
> >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
> >>> there
> >>>>>> might
> >>>>>>> be
> >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
> >>>>>>>>>>
> >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> >>>> default
> >>>>>>>> method,
> >>>>>>>>>> not an interface method?
> >>>>>>>>>>
> >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> >>>>> deprecated
> >>>>>>>>>> `registerCatalog`.
> >>>>>>>>>>
> >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> >>> mention
> >>>> the
> >>>>>>>> changed
> >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> >>>>> other
> >>>>>>>>>> persisted catalog store) is used.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Jark
> >>>>>>>>>>
> >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
> >>> jinfeng1995@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
> >>>>>>>>>>>
> >>>>>>>>>>> @Ron
> >>>>>>>>>>>
> >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
> >>>> method,
> >>>>>> do
> >>>>>>> we
> >>>>>>>>>> need
> >>>>>>>>>>> to provide a default implementation?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, we will provide a default
> >>> InMemoryCatalogStoreFactory to
> >>>>>>> create
> >>>>>>>> an
> >>>>>>>>>>> InMemoryCatalogStore.
> >>>>>>>>>>>
> >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
> >>> initializing
> >>>>> it,
> >>>>>>>> whether
> >>>>>>>>>> we
> >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, in the current design, catalogs are stored as
> >>> snapshots,
> >>>>> and
> >>>>>>>> once
> >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
> >>>>>> Catalog>
> >>>>>>>>>>> catalogs.
> >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
> >>> the
> >>>>>> primary
> >>>>>>>>>> source
> >>>>>>>>>>> for obtaining the corresponding Catalog.
> >>>>>>>>>>>
> >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
> >>>>>>>>>>> `catalog.store.path`?
> >>>>>>>>>>>
> >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
> >>> seem a
> >>>>> bit
> >>>>>>>>>> strange. I
> >>>>>>>>>>> will update the FLIP.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> @Shammon
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you for the review. I have made the necessary
> >>>>> corrections.
> >>>>>>>>>>> Regarding the modifications made to the Public Interface,
> >>> I
> >>>>> have
> >>>>>>> also
> >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Feng
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> >>>> zjureel@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi feng,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for updating, I have some minor comments
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
> >>>>>> `Public
> >>>>>>>>>>>> Interfaces`, it is not a public interface.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
> >>> and
> >>>>>>>>>>>> `CatalogStoreFactory`
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
> >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
> >>>>> should
> >>>>>> be
> >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
> >>>>>>>>>>>> catalogStore.get(catalogName);`
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
> >>>> ron9.liu@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Feng
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
> >>>> useful
> >>>>>> for
> >>>>>>>>>> catalog
> >>>>>>>>>>>>> management.
> >>>>>>>>>>>>> I have some small questions:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1. Regarding the
> >>> CatalogStoreFactory#createCatalogStore
> >>>>>> method,
> >>>>>>>> do we
> >>>>>>>>>>>> need
> >>>>>>>>>>>>> to provide a default implementation?
> >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
> >>>> initializing
> >>>>>> it,
> >>>>>>>>>> whether
> >>>>>>>>>>> we
> >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
> >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
> >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
> >>> them to
> >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Ron
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
> >>> 21:19写道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi yuxia
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
> >>>>> register
> >>>>>>> the
> >>>>>>>>>>> Catalog,
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>> initialize it and open it. right?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
> >>>>>> original
> >>>>>>>>>> CREATE
> >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
> >>>>>>>>>> registerCatalog
> >>>>>>>>>>> so
> >>>>>>>>>>>>>> that parameter validation can be performed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
> >>>>>>> reflected
> >>>>>>>> in
> >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
> >>>>> catalog
> >>>>>>>>>>>>> configurations,
> >>>>>>>>>>>>>> only initialization is required in getCatalog.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
> >>>>>>>> luoyuxia@alumni.sjtu.edu.cn
> >>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Feng.
> >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
> >>>>>>>> initialization*.
> >>>>>>>>>>> If
> >>>>>>>>>>>>> i'm
> >>>>>>>>>>>>>>> wrong, please correct me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
> >>>> access
> >>>>>> the
> >>>>>>>>>>> catalog,
> >>>>>>>>>>>>> then
> >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
> >>>>> Changes,
> >>>>>>>> once we
> >>>>>>>>>>>>>> register
> >>>>>>>>>>>>>>> the Catalog,
> >>>>>>>>>>>>>>> we initialize it and open it. right?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Yuxia
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ----- 原始邮件 -----
> >>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> >>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
> >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> >>>>>> Catalog
> >>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>> and asynchronous registration
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> One of the major changes is that current design
> >>> will
> >>>>>>> provide
> >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
> >>>> of a
> >>>>>>>> cache,
> >>>>>>>>>>> which
> >>>>>>>>>>>>>> means
> >>>>>>>>>>>>>>> once it has been initialized, any changes done by
> >>>> other
> >>>>>>>> sessions
> >>>>>>>>>>> will
> >>>>>>>>>>>>> not
> >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
> >>>>>> further
> >>>>>>>>>>>> improvement.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> >>>>>>>> jinfeng1995@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
> >>>>>> progress
> >>>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
> >>> Shammon
> >>>> FY,
> >>>>>>>> ShengKai
> >>>>>>>>>>> Fang
> >>>>>>>>>>>>>> and I
> >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
> >>>>>> solution
> >>>>>>>> for
> >>>>>>>>>>> Flink
> >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
> >>>> have
> >>>>>>>> updated
> >>>>>>>>>> the
> >>>>>>>>>>>>> final
> >>>>>>>>>>>>>>>> solution in FLIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 1.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
> >>>>>>>> configuration
> >>>>>>>>>>>> similar
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> TableDescriptor.
> >>>>>>>>>>>>>>>> 2.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
> >>>>>>>>>>> storeCatalog(String
> >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
> >>>>>> CatalogDescriptor
> >>>>>>>>>>>>>>> getCatalog(String)
> >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
> >>>> Catalog
> >>>>>>>> instance.
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>> way,
> >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
> >>> saving
> >>>>> and
> >>>>>>>>>>> retrieving
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> configurations without having to initialize
> >>>>>> catalogs.
> >>>>>>>>>>>>>>>> 3.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The default registerCatalog(String
> >>> catalogName,
> >>>>>>> Catalog
> >>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
> >>>>>>> deprecated.
> >>>>>>>>>>>>>>>> 4.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> A new function registerCatalog(String
> >>>> catalogName,
> >>>>>>>>>>>>> CatalogDescriptor
> >>>>>>>>>>>>>>>> catalog) will be added to serve as the
> >>> default
> >>>>>>>> registration
> >>>>>>>>>>>>> function
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> catalogs in CatalogManager.
> >>>>>>>>>>>>>>>> 5.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
> >>>> will
> >>>>>>>> remain
> >>>>>>>>>>>>> unchanged
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> save initialized catalogs.This means that
> >>>> deletion
> >>>>>>>>>> operations
> >>>>>>>>>>>> from
> >>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>> session won't synchronize with other
> >>> sessions.
> >>>>>>>>>>>>>>>> 6.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> To support multi-session synchronization
> >>>> scenarios
> >>>>>> for
> >>>>>>>>>>> deletions
> >>>>>>>>>>>>>> later
> >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
> >>>>>>>>>>> configurable.There
> >>>>>>>>>>>>> may
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>> three possible situations:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
> >>>>>>>> automatically
> >>>>>>>>>>> clear
> >>>>>>>>>>>>>>>> long-unused catalogs.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
> >>>>>> getCatalog
> >>>>>>>>>>> creates a
> >>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>> instance.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is the document for discussion:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is the final proposal document:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you very much for your attention and
> >>>>> suggestions
> >>>>>> on
> >>>>>>>> this
> >>>>>>>>>>>>> FLIP. A
> >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
> >>> on
> >>>>> the
> >>>>>>>> entire
> >>>>>>>>>>>> design
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> organizing offline discussions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
> >>>> about
> >>>>>>> this
> >>>>>>>> FLIP
> >>>>>>>>>>>>> please
> >>>>>>>>>>>>>>> feel
> >>>>>>>>>>>>>>>> free to share.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
> >>>>>>>>>>> <jing@ververica.com.invalid
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
> >>> to
> >>>> me.
> >>>>>> We
> >>>>>>>> could
> >>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
> >>>> common
> >>>>>>>> built-in
> >>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
> >>> every
> >>>>>>>>>> CatalogSotre
> >>>>>>>>>>>>>>>>> implementation has to take care of it.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> >>>>>>>>>> jinfeng1995@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Jing,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for your reply.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> >>>>> suggest
> >>>>>>> you
> >>>>>>>>>>>>> completely
> >>>>>>>>>>>>>>> walk
> >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
> >>> the
> >>>>> same
> >>>>>>>> time, I
> >>>>>>>>>>>>> greatly
> >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
> >>>> thoroughly
> >>>>>>>> checked
> >>>>>>>>>> the
> >>>>>>>>>>>>>> entire
> >>>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>> and made corrections to these issues.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
> >>> FLIP
> >>>>>>> design,
> >>>>>>>>>>>>>>> CatalogManager
> >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
> >>> being
> >>>>>>>> configured.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
> >>> was
> >>>> not
> >>>>>>>> necessary
> >>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
> >>>> catalogs
> >>>>>> to
> >>>>>>>> store
> >>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> instances.
> >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
> >>>>>>> Therefore, I
> >>>>>>>>>>>> modified
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> part
> >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
> >>> Catalog>
> >>>>>>> catalogs
> >>>>>>>>>> from
> >>>>>>>>>>>>>>>>>> CatalogManager.
> >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
> >>> serve
> >>>> as
> >>>>>> the
> >>>>>>>>>> default
> >>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
> >>>>>>>> functionality of
> >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
> >>>>>>>>>>>>>>>>>> The previous plan that kept
> >>>>>>>> Map<String,Catalog>catalogs has
> >>>>>>>>>>>> been
> >>>>>>>>>>>>>>> moved
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> Rejected Alternatives.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> >>>>>>>>>>>>> <jing@ververica.com.invalid
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There are still many places contain
> >>>>> inconsistent
> >>>>>>>> content,
> >>>>>>>>>>>> e.g.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
> >>> used.
> >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
> >>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
> >>> not
> >>>>> tell
> >>>>>>>> what the
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> doing.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> >>>>> suggest
> >>>>>>> you
> >>>>>>>>>>>> completely
> >>>>>>>>>>>>>>> walk
> >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
> >>> issues.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
> >>> mean
> >>>>> that
> >>>>>>> you
> >>>>>>>> will
> >>>>>>>>>>>> build
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
> >>> is a
> >>>>> very
> >>>>>>>>>> different
> >>>>>>>>>>>>>> design
> >>>>>>>>>>>>>>>>>> concept
> >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
> >>> I am
> >>>>> not
> >>>>>>>>>> mistaken,
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
> >>>> work
> >>>>>>>> without
> >>>>>>>>>>>> Optional
> >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
> >>>>> reason
> >>>>>>>> why I
> >>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
> >>>>> Optional
> >>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> correct.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> >>>>>>>>>>>>> jinfeng1995@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Jing
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
> >>>> FLIP
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
> >>>> review. I
> >>>>>>> have
> >>>>>>>>>>> already
> >>>>>>>>>>>>> made
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> relevant changes.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
> >>> the
> >>>>>>>>>> multi-instance
> >>>>>>>>>>>>>>>> scenarios, I
> >>>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
> >>> multiple
> >>>>>>>> scenarios
> >>>>>>>>>> with
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
> >>>>> remove
> >>>>>>>>>>> 'Map<String,
> >>>>>>>>>>>>>>>> Catalog>'
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
> >>>>>>>>>> InMemoryCatalogStore.
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>>> way,
> >>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
> >>>>> achieve
> >>>>>>>>>>>>> multi-instance
> >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
> >>>> think?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> >>>>>>>>>>>>>>> <jing@ververica.com.invalid
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
> >>> are
> >>>>>> still
> >>>>>>>> some
> >>>>>>>>>>> NIT
> >>>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>>>>> like:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
> >>>> has
> >>>>>> been
> >>>>>>>> used
> >>>>>>>>>> as
> >>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
> >>> example.
> >>>> It
> >>>>>>>> should be
> >>>>>>>>>>>> fine
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
> >>> after
> >>>> you
> >>>>>>>> submit
> >>>>>>>>>> the
> >>>>>>>>>>>> PR.
> >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
> >>>> somewhere
> >>>>> in
> >>>>>>> the
> >>>>>>>>>>>> rejected
> >>>>>>>>>>>>>>>> section
> >>>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
> >>> it
> >>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
> >>>>>>>> multi-instance
> >>>>>>>>>>>>>>> scenarios,
> >>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
> >>> behaviour.
> >>>> If
> >>>>>>>>>> Map<String,
> >>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
> >>>> once
> >>>>>> the
> >>>>>>>>>> related
> >>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
> >>> another
> >>>>>>>> instance.
> >>>>>>>>>> Did
> >>>>>>>>>>> I
> >>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>>> something?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
> >>> Jin
> >>>> <
> >>>>>>>>>>>>>>> jinfeng1995@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
> >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> @Jing
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
> >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
> >>> name, I
> >>>>>> have
> >>>>>>>>>> updated
> >>>>>>>>>>> it
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> documentation.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
> >>> should
> >>>> be
> >>>>>>>>>> implemented
> >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
> >>>>>> alternative
> >>>>>>> 1
> >>>>>>>>>> turns
> >>>>>>>>>>>> out
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
> >>> this
> >>>>> part
> >>>>>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> content.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
> >>>> forward,
> >>>>>> the
> >>>>>>>>>> option 2
> >>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
> >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
> >>>>> offline,
> >>>>>>> we
> >>>>>>>>>> think
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>> 2
> >>>>>>>>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
> >>>>>>>> inconsistency
> >>>>>>>>>>>> issues.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
> >>>>> descriptions
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>>> content.
> >>>>>>>>>>>>>>>> Would
> >>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>> to clean them up?
> >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
> >>>> document
> >>>>>> and
> >>>>>>>>>>>> appreciate
> >>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>> suggestions.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
> >>> in
> >>>>>>>> `Rejected
> >>>>>>>>>>>>>>> Alternatives`
> >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
> >>> `Rejected
> >>>>>>>>>> Alternatives`.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best
> >>>>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
> >>> Shammon
> >>>>> FY
> >>>>>> <
> >>>>>>>>>>>>>>> zjureel@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
> >>>>>> `Proposed
> >>>>>>>>>>> Changes`,
> >>>>>>>>>>>>> can
> >>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>> put
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
> >>>>>>>> Alternatives`? I
> >>>>>>>>>>> think
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>> help
> >>>>>>>>>>>>>>>>>>>> us
> >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
> >>> Jing
> >>>> Ge
> >>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
> >>> am
> >>>>>> still
> >>>>>>>>>> confused
> >>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>> completely
> >>>>>>>>>>>>>>>>>>>>>>> reading
> >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
> >>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
> >>>>> registerCatalog()
> >>>>>>> and
> >>>>>>>>>>>>>> addCatalog()
> >>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>>>>>> big
> >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
> >>>> One
> >>>>> of
> >>>>>>>> them is
> >>>>>>>>>>>>>>> responsible
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
> >>>>>> persistCatalog()?
> >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
> >>>> Map<String,
> >>>>>>>> Catalog>
> >>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>> as a
> >>>>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
> >>> data
> >>>>>>>> persistence.
> >>>>>>>>>> I
> >>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>>> describing
> >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
> >>>>> clearly
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>> Some
> >>>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
> >>>> i.e.
> >>>>>> data
> >>>>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
> >>>> instance
> >>>>>>>> should be
> >>>>>>>>>>>> found
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
> >>>>>> initialized
> >>>>>>>> or it
> >>>>>>>>>>> will
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>>>>> initialized)
> >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
> >>>>>>> consistency
> >>>>>>>> will
> >>>>>>>>>>> be
> >>>>>>>>>>>>>> taken
> >>>>>>>>>>>>>>>> care
> >>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> while
> >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
> >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
> >>>>>> forward,
> >>>>>>>> the
> >>>>>>>>>>>> option
> >>>>>>>>>>>>> 2
> >>>>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
> >>>>>> because,
> >>>>>>>> afaiu,
> >>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
> >>>>> solved
> >>>>>>>> yet. Do
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>> want
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> propose
> >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
> >>>> suggestions
> >>>>>> for
> >>>>>>>> both
> >>>>>>>>>> of
> >>>>>>>>>>>>> them?
> >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
> >>>> there
> >>>>>> are
> >>>>>>>> some
> >>>>>>>>>>>>>>> inconsistent
> >>>>>>>>>>>>>>>>>>>>> descriptions
> >>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
> >>>> clean
> >>>>>>> them
> >>>>>>>> up?
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
> >>> Feng
> >>>>>> Jin <
> >>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
> >>> and
> >>>> I
> >>>>>>>> completely
> >>>>>>>>>>>> agree
> >>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> view.
> >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
> >>>>>> complicated
> >>>>>>>> the
> >>>>>>>>>>> whole
> >>>>>>>>>>>>>> issue.
> >>>>>>>>>>>>>>>>> First
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> foremost,
> >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
> >>>> persistence
> >>>>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> Catalog's
> >>>>>>>>>>>>>>>>>>>> Configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
> >>>>>> persistence
> >>>>>>>> for
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>> Configuration,
> >>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
> >>> to
> >>>> the
> >>>>>>>> Catalog
> >>>>>>>>>>>>>> interface.
> >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
> >>> Catalog
> >>>>>>>> instance
> >>>>>>>>>> to a
> >>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
> >>>>>>>> implementation
> >>>>>>>>>> will
> >>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
> >>>>>>>>>>>>>>>>>>>>>>>>> /**
> >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
> >>> the
> >>>>>>>> properties
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>> object.
> >>>>>>>>>>>>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
> >>> the
> >>>>>>>> properties
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>> object
> >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
> >>>>>>>> UnsupportedOperationException if
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> implementing
> >>>>>>>>>>>>>>>>>>>>> class
> >>>>>>>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>>>>>>> not override
> >>>>>>>>>>>>>>>>>>>>>>>>> * the default
> >>> implementation
> >>>> of
> >>>>>> this
> >>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> */
> >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
> >>>>>>>> toProperties()
> >>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>> throw new
> >>>>>>>>>>>> UnsupportedOperationException("Please
> >>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>> toProperties
> >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
> >>>> follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
> >>> configured a
> >>>>>>>>>> CatalogStore,
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> toConfiguration()
> >>>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
> >>> registering a
> >>>>>>> Catalog
> >>>>>>>>>>>> instance
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>> catalogName,
> >>>>>>>> Catalog
> >>>>>>>>>>>>> catalog).
> >>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
> >>>> Map<String,
> >>>>>>>> String>
> >>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> saved
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
> >>>>>> instances
> >>>>>>>> do
> >>>>>>>>>> not
> >>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> method,
> >>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
> >>>>>>>> CatalogStore,
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> toConfiguration()
> >>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
> >>>>>>> consistency
> >>>>>>>> with
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>>>>> process.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
> >>>>>> Catalog>
> >>>>>>>>>>> catalogs
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
> >>>>>> conflicts
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
> >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
> >>>>> itself
> >>>>>>> and
> >>>>>>>>>>> provide
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> dedicated
> >>>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
> >>> loading,
> >>>>> such
> >>>>>>> as
> >>>>>>>>>>>>>>>>> ConfigurationCatalog
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> public class
> >>> ConfigurationCatalog
> >>>>>>>> implements
> >>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>> ConfigurationCatalog(Map<String,
> >>>>>>> String>
> >>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
> >>> the
> >>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
> >>> 10:03 AM
> >>>>>> Shammon
> >>>>>>>> FY <
> >>>>>>>>>>>>>>>>>> zjureel@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
> >>> the
> >>>>>>>> functionality
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> `CatalogStore`,
> >>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
> >>>> multiple
> >>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> >>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
> >>>>>> implementation
> >>>>>>> in
> >>>>>>>>>> Trino
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> same,
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
> >>> only
> >>>>> has
> >>>>>>> one
> >>>>>>>>>>>>>>> `createCatalog`,
> >>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>> save
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
> >>>>>> `CatalogStore`.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
> >>>>>> another
> >>>>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
> >>> `Map<String,
> >>>>>>> Catalog>
> >>>>>>>>>>>> catalogs`
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> `CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
> >>>> mentioned
> >>>>>>>> before,
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> `Map<String,
> >>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
> >>>>>>>> `CatalogManager`.
> >>>>>>>>>> How
> >>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>> saving
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
> >>> catalogs`
> >>>>> and
> >>>>>>>>>>>> `CatalogStore
> >>>>>>>>>>>>>>>>>>> catalogStore`
> >>>>>>>>>>>>>>>>>>>>>>>> together
> >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
> >>> added
> >>>> in
> >>>>>>>>>>>>> `CatalogManager`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>> 5:22 PM
> >>>>> Feng
> >>>>>>>> Jin <
> >>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
> >>> I
> >>>> am
> >>>>>> very
> >>>>>>>> sorry
> >>>>>>>>>>> for
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> misunderstanding
> >>>>>>>>>>>>>>>>>>>>>>>>> caused
> >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
> >>>>> original
> >>>>>>>>>>> discussion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> >>>>>> pre-discussion
> >>>>>>>> [1]
> >>>>>>>>>> for
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
> >>> such a
> >>>>>>>> discussion
> >>>>>>>>>>>> before.
> >>>>>>>>>>>>>>>>> However,
> >>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> designing
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
> >>> that
> >>>>> the
> >>>>>>>> logic of
> >>>>>>>>>>>>>>>>> CatalogManager
> >>>>>>>>>>>>>>>>>>>> itself
> >>>>>>>>>>>>>>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
> >>>> cannot
> >>>>>>> only
> >>>>>>>>>>> persist
> >>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
> >>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
> >>>> Catalogs,
> >>>>>>>> because
> >>>>>>>>>> in
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> end
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> save
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
> >>>>> corresponding
> >>>>>> to
> >>>>>>>> the
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>> instances.
> >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
> >>>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
> >>>>>>> CatalogStore
> >>>>>>>>>>>> interface
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
> >>> this
> >>>>>> part, I
> >>>>>>>> also
> >>>>>>>>>>> took
> >>>>>>>>>>>>>>>>> inspiration
> >>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>> Trino's
> >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> >>>> handling
> >>>>>>>> catalog
> >>>>>>>>>> with
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>> looks a
> >>>>>>>>>>>>>>>>>>>>>>>>> little
> >>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> >>>> suggestion. I
> >>>>>>> will
> >>>>>>>>>>> provide
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> detailed
> >>>>>>>>>>>>>>>>>>>>>> description
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
> >>> part,
> >>>>> and
> >>>>>>> add
> >>>>>>>> it
> >>>>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
> >>>>>>>> `addCatalog`
> >>>>>>>>>>> still
> >>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>> confused
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
> >>>>>> catalog
> >>>>>>> to
> >>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>> renaming
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
> >>> very
> >>>>>>> personal
> >>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> decide
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> take
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
> >>>>> intuitive
> >>>>>>> to
> >>>>>>>> me.
> >>>>>>>>>> I
> >>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>> see
> >>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>> problem
> >>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
> >>>>> statement,
> >>>>>>>> which
> >>>>>>>>>>>> method
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> called?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> >>>>>> `addCatalog`?
> >>>>>>>> I'm
> >>>>>>>>>>>>> wondering
> >>>>>>>>>>>>>>>>> whether
> >>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
> >>>>>> stement.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
> >>>> original
> >>>>>>>> design
> >>>>>>>>>> was
> >>>>>>>>>>> to
> >>>>>>>>>>>>> add
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> directly
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
> >>> would
> >>>>>>> disrupt
> >>>>>>>> the
> >>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>> logic.
> >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
> >>>>>>>>>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
> >>>>>>>> configuration to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> initialize
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
> >>> the
> >>>>> same
> >>>>>>> time
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
> >>> neccessary
> >>>> to
> >>>>>>>> provide a
> >>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>> implmentation
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> >>>>> `CatalogStoreFactory`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
> >>>>>> otherwise
> >>>>>>> we
> >>>>>>>>>> would
> >>>>>>>>>>>>> need
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> introduce
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>> additional
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
> >>>> configuration
> >>>>>> for
> >>>>>>>> lazy
> >>>>>>>>>>>>> loading.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> >>>>>>> registration
> >>>>>>>> for
> >>>>>>>>>>>>> catalog.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
> >>>>>>> registerCatalog(String
> >>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
> >>>>>> interface
> >>>>>>>>>> because
> >>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
> >>>>>>>> initialization
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
> >>>>>>>> inaccurate. It
> >>>>>>>>>>>>> should
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
> >>>>>>>> registration. I
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> title
> >>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>>> 4:27 PM
> >>>>>>>> Shammon
> >>>>>>>>>> FY <
> >>>>>>>>>>>>>>>>>>> zjureel@gmail.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
> >>>>>>>> `registerCatalog`
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
> >>>>> method
> >>>>>>>> named
> >>>>>>>>>>>>>> `addCatalog`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> >>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
> >>> the
> >>>>>> memory
> >>>>>>>>>> catalog
> >>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>> `cache`,
> >>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
> >>>>>> exposed
> >>>>>>> to
> >>>>>>>>>>> users.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> >>>>>> pre-discussion
> >>>>>>>> [1]
> >>>>>>>>>> for
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>> Please
> >>>>>>>>>>>>>>>>>>>>>>> correct
> >>>>>>>>>>>>>>>>>>>>>>>>> me
> >>>>>>>>>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
> >>>>> conclusion
> >>>>>>> of
> >>>>>>>> that
> >>>>>>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
> >>>>>> interface
> >>>>>>>> and
> >>>>>>>>>>>>> implement
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>>>>>>>>> stores
> >>>>>>>>>>>>>>>>>>>>>>>>>>> such
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
> >>>> external
> >>>>>>>> system.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
> >>>>> between
> >>>>>>> the
> >>>>>>>>>>> current
> >>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>> design
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
> >>> of
> >>>>> the
> >>>>>>>>>> discussion
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>> thread
> >>>>>>>>>>>>>>>>>> [1] ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>>>> 3:41 PM
> >>>>>>>> yuxia <
> >>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
> >>> driving
> >>>>> this
> >>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> >>>>> handling
> >>>>>>>> catalog
> >>>>>>>>>>> with
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>>> looks a
> >>>>>>>>>>>>>>>>>>>>>>>>>> little
> >>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
> >>>>> it'll
> >>>>>> be
> >>>>>>>>>> better
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>> explain
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> java
> >>>>>>>>>>>>>>>>>>>>>>>>> doc
> >>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
> >>> give a
> >>>>>> brief
> >>>>>>>>>> example
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
> >>> name
> >>>>>>>> `addCatalog`
> >>>>>>>>>>>> still
> >>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>> confused
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
> >>>> storing
> >>>>>>>> catalog to
> >>>>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
> >>> It's
> >>>>> very
> >>>>>>>> personal
> >>>>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>> decide
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> take
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
> >>>>>>> statement,
> >>>>>>>>>> which
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> called?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> >>>>>>>> `addCatalog`?
> >>>>>>>>>> I'm
> >>>>>>>>>>>>>>> wondering
> >>>>>>>>>>>>>>>>>>> whether
> >>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
> >>> SQL
> >>>>>>>> stement.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
> >>>> neccessary
> >>>>> to
> >>>>>>>>>> provide a
> >>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>> implmentation
> >>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> >>>>>>> `CatalogStoreFactory`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> >>>>>>>> registration
> >>>>>>>>>> for
> >>>>>>>>>>>>>> catalog.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
> >>> catalog
> >>>>>> with
> >>>>>>>> CREATE
> >>>>>>>>>>>>>> CATALOG,
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> >>>> used
> >>>>>> by
> >>>>>>>>>> default.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
> >>>>> registration
> >>>>>> is
> >>>>>>>> the
> >>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>> behavior,
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
> >>> to
> >>>>>>>> synchronous
> >>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>>>> before?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
> >>>>> `addCatalog`
> >>>>>>> and
> >>>>>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
> >>>>>>>> registration,
> >>>>>>>>>> it
> >>>>>>>>>>>> may
> >>>>>>>>>>>>>> well
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
> >>>> but
> >>>>>> then
> >>>>>>>> the
> >>>>>>>>>>>>> following
> >>>>>>>>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>>>>>>> TABLE
> >>>>>>>>>>>>>>>>>>>>>>>>> statement
> >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
> >>> fail
> >>>>> to
> >>>>>>>> open.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
> >>>> change
> >>>>>>> which
> >>>>>>>>>>> should
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> highlighted
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
> >>> part.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
> >>>> asynchronous
> >>>>>>>>>>>> registration, I
> >>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> expect
> >>>>>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
> >>>> open
> >>>>>> or
> >>>>>>>>>> register
> >>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> background,
> >>>>>>>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
> >>> comments,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
> >>>> will
> >>>>> be
> >>>>>>>>>>> initialized
> >>>>>>>>>>>>> if
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
> >>>> initialization
> >>>>>>>> process
> >>>>>>>>>>> fails,
> >>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>> statements
> >>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
> >>> successfully."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
> >>> lazy
> >>>>>>>>>> initialization
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
> >>>>>>>>>> jinfeng1995@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
> >>>>>>> dev@flink.apache.org
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
> >>> 03日
> >>>> 下午
> >>>>>>>> 3:27:45
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
> >>>> 295:
> >>>>>>>> Support
> >>>>>>>>>>>>>> persistence
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
> >>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
> >>>> all
> >>>>>> for
> >>>>>>>> your
> >>>>>>>>>>>>> interest
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> DISCUSS.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
> >>> catalog
> >>>>>> with
> >>>>>>>> the
> >>>>>>>>>> same
> >>>>>>>>>>>>> name
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
> >>>> crucial
> >>>>>>>> point.
> >>>>>>>>>>> Based
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>>>> personal
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
> >>>>>> Map<String,
> >>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>> serve
> >>>>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
> >>> catalogs
> >>>>> and
> >>>>>>>> have
> >>>>>>>>>> the
> >>>>>>>>>>>>>> highest
> >>>>>>>>>>>>>>>>>>> priority.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
> >>> methods
> >>>>> that
> >>>>>>> can
> >>>>>>>>>> affect
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> >>> registerCatalog(String
> >>>>>>>>>> catalogName,
> >>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
> >>>>> catalog
> >>>>>>>> instance
> >>>>>>>>>>>> into
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> >>>> unregisterCatalog(String
> >>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> >>> the
> >>>>>> catalog
> >>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
> >>>>> Catalog>
> >>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
> >>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
> >>>> retrieves
> >>>>>> the
> >>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
> >>> Catalog>
> >>>>>>>> catalogs. If
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>> exist,
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
> >>>> corresponding
> >>>>>>>>>>> configuration
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
> >>> puts
> >>>>> the
> >>>>>>>>>>> initialized
> >>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> >>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
> >>> methods
> >>>>>> only
> >>>>>>>> modify
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
> >>>>>>> catalogName,
> >>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
> >>>>>>> properties
> >>>>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> catalogStore
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> checks
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
> >>>>>> catalogName
> >>>>>>>> with
> >>>>>>>>>> the
> >>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>> name.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> >>> removeCatalog(String
> >>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> >>> the
> >>>>>>> specified
> >>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> specified
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
> >>>>>>> catalogStore.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
> >>>> possible
> >>>>>>>> conflict
> >>>>>>>>>>>>>> scenarios:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
> >>> in
> >>>> the
> >>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>> Catalog>,
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>>>>>> catalogName,
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
> >>> directly
> >>>>> saved
> >>>>>>> to
> >>>>>>>> the
> >>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
> >>>>>>> Map<String,
> >>>>>>>>>>>> Catalog>,
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
> >>>>> catalog)
> >>>>>>>> method
> >>>>>>>>>>> will
> >>>>>>>>>>>>>> fail.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
> >>> the
> >>>>>>>>>>>> addCatalog(String
> >>>>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
> >>>> method
> >>>>>> can
> >>>>>>>>>> directly
> >>>>>>>>>>>>> save
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
> >>>>>>>>>> getCatalog(String
> >>>>>>>>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
> >>>> for
> >>>>>>>>>>> initialization
> >>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
> >>> already
> >>>>>> exists
> >>>>>>>> in
> >>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
> >>>>>>>>>>>>> unregisterCatalog(String
> >>>>>>>>>>>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
> >>>>>>>> corresponding to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>>>>>> catalogName
> >>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
> >>> confuse
> >>>>>> users
> >>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
> >>>>> catalog)`
> >>>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>>>>> `Map<String,
> >>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> `registerCatalog(String
> >>>>>>>> catalogName,
> >>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
> >>>> lead
> >>>>>> to
> >>>>>>>>>>>> confusion. I
> >>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>> changing
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
> >>> to
> >>>>>>>>>>> addCatalog(String
> >>>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
> >>> as
> >>>>>>>> previously
> >>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
> >>>>> `registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,Catalog
> >>>>>>>>>>>>>>>>>>>> catalog,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
> >>> method
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
> >>>> already
> >>>>> an
> >>>>>>>>>> instance,
> >>>>>>>>>>>>>> adding
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> "lazyInit"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
> >>>>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
> >>>> necessarily
> >>>>>>>> result in
> >>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>> initialization.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
> >>>> about
> >>>>>>>>>> encryption
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
> >>>>>>> necessary,
> >>>>>>>> but
> >>>>>>>>>>>>> perhaps
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> encryption
> >>>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
> >>> in
> >>>>> the
> >>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> actually
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
> >>> the
> >>>>>>>>>>> FileCatalogStore
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>> encrypt
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
> >>>>> while
> >>>>>>> the
> >>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>> require
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
> >>> the
> >>>>>>>>>>>>>> `MemoryCatalogStore`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
> >>> necessary
> >>>> to
> >>>>>>> have a
> >>>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
> >>>> saves
> >>>>>>>> Catalog
> >>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> memory.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
> >>> to
> >>>>>>>> implement
> >>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>> loading
> >>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
> >>> introduce
> >>>>>>>> additional
> >>>>>>>>>>> cache.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
> >>>>>>>> registration
> >>>>>>>>>>>> failed?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
> >>> critical
> >>>>>>> concern.
> >>>>>>>>>> When
> >>>>>>>>>>>>>>> executing
> >>>>>>>>>>>>>>>>> DDL,
> >>>>>>>>>>>>>>>>>>>> DQL,
> >>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> DML
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
> >>>> reference a
> >>>>>>>> specific
> >>>>>>>>>>>>> Catalog,
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
> >>> if it
> >>>>> has
> >>>>>>> not
> >>>>>>>>>> been
> >>>>>>>>>>>>>>>> initialized
> >>>>>>>>>>>>>>>>>> yet.
> >>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
> >>>>> fails,
> >>>>>>>> these
> >>>>>>>>>>>>>> statements
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> executed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
> >>>>>> Catalog>
> >>>>>>>>>> catalogs
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
> >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
> >>>> have
> >>>>>> the
> >>>>>>>> same
> >>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> name,
> >>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>>> correponding
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> >>>>> instances.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
> >>>> resolved
> >>>>> by
> >>>>>>>>>> referring
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>>>>>>>>>>> responses.
> >>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
> >>> the
> >>>>>>> Catalog
> >>>>>>>>>> that
> >>>>>>>>>>>> has
> >>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>> recently
> >>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
> >>>> for
> >>>>>>>> subsequent
> >>>>>>>>>>>> use.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
> >>>>>> 10:58 PM
> >>>>>>>>>> Xianxun
> >>>>>>>>>>>> Ye
> >>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
> >>> this
> >>>>>> Flip,
> >>>>>>>> I do
> >>>>>>>>>>>>> believe
> >>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> Flip
> >>>>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
> >>>> work
> >>>>>> for
> >>>>>>>> also
> >>>>>>>>>>>>> manages a
> >>>>>>>>>>>>>>> lot
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> catalogs,
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
> >>>> becomes
> >>>>>>> slow
> >>>>>>>>>>> because
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> loading a
> >>>>>>>>>>>>>>>>>>>>> number
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
> >>> catalogs in
> >>>>> the
> >>>>>>>> user's
> >>>>>>>>>>>> Flink
> >>>>>>>>>>>>>> SQL
> >>>>>>>>>>>>>>>>>> through
> >>>>>>>>>>>>>>>>>>>>>> regular
> >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
> >>>>> catalogs
> >>>>>>> to
> >>>>>>>>>>> improve
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> speed
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
> >>> flip-295, I
> >>>>>> have
> >>>>>>>> some
> >>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
> >>>>> catalog
> >>>>>>> with
> >>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>> CATALOG,
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> >>>> used
> >>>>>> by
> >>>>>>>>>> default.
> >>>>>>>>>>>>> What
> >>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
> >>>>> implement
> >>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>> registration?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
> >>> same
> >>>>>>>> question
> >>>>>>>>>> with
> >>>>>>>>>>>>>>> Shammon’s
> >>>>>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>>>>>>>>>> comment.
> >>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> >>>>> catalogs
> >>>>>>> and
> >>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>> catalogstore
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
> >>> have
> >>>> the
> >>>>>>> same
> >>>>>>>>>>> catalog
> >>>>>>>>>>>>>> name,
> >>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>> correponding
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> >>>>> instances.
> >>>>>>> For
> >>>>>>>>>>> example:
> >>>>>>>>>>>>>>>>> catalog1->
> >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
> >>> catalog1 ->
> >>>>>>>> hiveCatalog
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> catalogstore.
> >>>>>>>>>>>>>>>>>>>>>>> Would
> >>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
> >>> should
> >>>>>> define
> >>>>>>>>>>> `catalogs`
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> `catalogstore`
> >>>>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
> >>> 11:13，Hang
> >>>>>> Ruan
> >>>>>>> <
> >>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 写道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Leonard Xu <xb...@gmail.com>.
Hi, Feng
Sorry for reply late, but I’ve some comments about the FLIP
1. The introduced Public class CatalogDescriptor seems missed some necessary component
a) How to construct a CatalogDescriptor ?
b) How to visit the fields ? Could we use Configuration instead of Map<String, String> ?
c) Could we offer a built-in factory method to build a CatalogDescriptor like
public static CatalogDescriptor of(String catalogName, Configuration configuration)
2. The FLIP said “By default, there are two built-in CatalogStores available: the In-Memory CatalogStore and the File CatalogStore” ,
Do we have plan to offer a default CatalogStore if user didn’t config this? IIUC, users can obtains the benefits from lazy catalog initialization If
we have a default catalogstore even it is in-memory.
Best,
Leonard
> On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
>
> Hi everyone,
>
> Thank you everyone for your valuable input. If there are no further
> questions or concerns about the FLIP[1], I would like to start voting
> tomorrow (6/7).
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>
>
> Best,
> Feng
>
>
> On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com> wrote:
>
>> Hi Samrat,
>>
>> Thanks for your advice.
>>
>>> 1. The createCatalog method does not mention any exceptions being
>> thrown.
>>
>> CreateCatalog will throw CatalogException like registerCatalog. As
>> CatalogException is a RuntimeException,
>> there is no explicit declaration of throwing Exceptions in CatalogManager
>> and TableEnvironment.
>> To avoid misunderstandings, I have added the "throw CatalogException" flag
>> to the createCatalog method definition of CatalogStore.
>>
>>> 2. Could you please provide an exhaustive list of the supported kinds?
>>
>> Sure, the documentation now includes the configuration of the built-in
>> CatalogStore as well as how to configure a custom CatalogStore.
>>
>>
>> Best,
>> Feng
>>
>>
>> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com> wrote:
>>
>>> Hi Feng,
>>>
>>> Thank you for providing the proposal. I believe this feature will be
>>> highly
>>> valuable.
>>>
>>> I have a couple of inquiries:
>>>
>>> 1. According to the documentation [1], the createCatalog method does not
>>> mention any exceptions being thrown. However, I would like to confirm if
>>> it
>>> is always true that there will be no failures in all scenarios. Please let
>>> me know if there is any additional information I may have missed.
>>>
>>> 2. Regarding the registration process using the `table.catalog-store.kind`
>>> configuration, could you please provide an exhaustive list of the
>>> supported
>>> kinds?
>>> It would be great to have a comprehensive understanding of the options
>>> available.
>>>
>>> Bests,
>>> Samrat
>>>
>>> [1]
>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>>>
>>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
>>>
>>>> Hi Hang and Jark
>>>>
>>>> Thank you very much for your reply.
>>>>
>>>> @Jark
>>>>> 1. Could you move the CatalogStore registration API to the "Public
>>>> Interface" section?
>>>>
>>>> Indeed, it is more reasonable.
>>>>
>>>>> 2. We should prefix "table." for the CatalogStore configuration.
>>>>
>>>> Sure, the names that you provided are indeed more logical and easier
>>> to
>>>> understand.
>>>>
>>>>> 3. About the open/close method in CatalogStoreFactory.
>>>>
>>>> I also agree with the proposed requirement scenario and design. I have
>>>> already made modifications to the interface.
>>>>
>>>> @Hang
>>>>
>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or close
>>>> the resource.
>>>>
>>>> I have already added the missing method.
>>>>
>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>>>>
>>>> Sorry for the misspelling, I updated this part.
>>>>
>>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
>>>> to init or close its resource
>>>>
>>>> This is indeed a scenario that needs to be considered, and I have added
>>>> relevant methods accordingly.
>>>> Additionally, I have included some relevant description information in
>>> the
>>>> documentation to help others understand
>>>> why the open/close methods need to be added.
>>>>
>>>>
>>>> Best,
>>>> Feng
>>>>
>>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
>>>>
>>>>> Hi Jing,
>>>>>
>>>>> Thank you for the update.
>>>>>
>>>>> 1. Could you move the CatalogStore registration API to the "Public
>>>>> Interface" section?
>>>>> "Proposed Changes" is more like a place to describe the implementation
>>>>> details.
>>>>>
>>>>> 2. We should prefix "table." for the CatalogStore configuration.
>>> Besides,
>>>>> the config key
>>>>> name should be hierarchical[1]. Therefore, it may be better to use:
>>>>> "table.catalog-store.kind"
>>>>> "table.catalog-store.file.path"
>>>>>
>>>>> 3. I think Hang's suggestions make sense.
>>>>>
>>>>> Others look good to me.
>>>>>
>>>>> Best,
>>>>> Jark
>>>>>
>>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
>>> wrote:
>>>>>
>>>>>> Hi, Feng.
>>>>>>
>>>>>> Thanks for the update.
>>>>>> The current design looks good to me. I have some minor comments.
>>>>>>
>>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
>>> close
>>>>> the
>>>>>> resource. For example, when storing the information in MySQL, the
>>> store
>>>>>> needs to open and close the connections.
>>>>>>
>>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>>>>>>
>>>>>> 3. About the usage in the sql gateway.
>>>>>> Considering the usage in sql gateway, the sql gateway may create a
>>>>>> CatalogStore for each session.
>>>>>> If we are using the MySqlCatalogStore, there would be so many
>>>>> connections.
>>>>>> How can we reuse the connection among these sessions?
>>>>>> I think sql gateway need to maintain a connection pool in
>>>>>> the CatalogStoreFactory and each session get its own connection from
>>>> the
>>>>>> pool when it is created.
>>>>>> Then the `CatalogStoreFactory` may need the `open`/`close` methods
>>> to
>>>>> init
>>>>>> or close its resource.
>>>>>> Or is there a better way?
>>>>>>
>>>>>> Best,
>>>>>> Hang
>>>>>>
>>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
>>>>>>
>>>>>>> Thanks Jingsong.
>>>>>>>
>>>>>>>> Just naming, maybe `createCatalog` in TableEnv
>>>>>>>
>>>>>>> +1 For this, I have already updated FLIP.
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Feng
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
>>> jingsonglee0@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Feng,
>>>>>>>>
>>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
>>>>>>>> functions are converted to createxxx from registerxxx.
>>>>>>>>
>>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1995@gmail.com
>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi jark, thanks for your suggestion.
>>>>>>>>>
>>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
>>> the
>>>>>>>>> CatalogStore should be immutable once TableEnv is created.
>>>>> Otherwise,
>>>>>>>> there
>>>>>>>>> might be some data inconsistencies when CatalogStore is
>>> changed.
>>>>>>>>>
>>>>>>>>> Yes, We should initialize the CatalogStore when creating the
>>>>>> TableEnv.
>>>>>>>>> Therefore, my current proposal is to add a method to configure
>>>> the
>>>>>>>>> CatalogStore in EnvironmentSettings.
>>>>>>>>>
>>>>>>>>> // Initialize a catalog Store
>>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
>>>>>>>>>
>>>>>>>>> // set up the Table API
>>>>>>>>> final EnvironmentSettings settings =
>>>>>>>>>
>>> EnvironmentSettings.newInstance().inBatchMode()
>>>>>>>>> .withCatalogStore(catalogStore)
>>>>>>>>> .build();
>>>>>>>>>
>>>>>>>>> final TableEnvironment tableEnv =
>>>>>>>> TableEnvironment.create(settings);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
>>>> default
>>>>>>>> method,
>>>>>>>>> not an interface method?
>>>>>>>>>
>>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
>>> that
>>>> as
>>>>> a
>>>>>>> new
>>>>>>>>> interface, the CatalogStoreFactory should not have a default
>>>> method
>>>>>> but
>>>>>>>> an
>>>>>>>>> interface method. I have already modified the interface.
>>>>>>>>>
>>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
>>>>>>>>> deprecated`registerCatalog`.
>>>>>>>>>> 4. In the "Compatibility" section, would be better to
>>> mention
>>>> the
>>>>>>>> changed
>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
>>>> other
>>>>>>>>> persisted catalog store) is used.
>>>>>>>>>
>>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1].
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Feng
>>>>>>>>>
>>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Feng,
>>>>>>>>>>
>>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
>>>>>>>>>> The current design looks pretty good to me. I just have some
>>>>> minor
>>>>>>>>>> comments.
>>>>>>>>>>
>>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
>>> the
>>>>>>>> CatalogStore
>>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
>>> there
>>>>>> might
>>>>>>> be
>>>>>>>>>> some data inconsistencies when CatalogStore is changed.
>>>>>>>>>>
>>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
>>>> default
>>>>>>>> method,
>>>>>>>>>> not an interface method?
>>>>>>>>>>
>>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
>>>>> deprecated
>>>>>>>>>> `registerCatalog`.
>>>>>>>>>>
>>>>>>>>>> 4. In the "Compatibility" section, would be better to
>>> mention
>>>> the
>>>>>>>> changed
>>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
>>>>> other
>>>>>>>>>> persisted catalog store) is used.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Jark
>>>>>>>>>>
>>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
>>> jinfeng1995@gmail.com>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi ， thanks all for reviewing the flip.
>>>>>>>>>>>
>>>>>>>>>>> @Ron
>>>>>>>>>>>
>>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
>>>> method,
>>>>>> do
>>>>>>> we
>>>>>>>>>> need
>>>>>>>>>>> to provide a default implementation?
>>>>>>>>>>>
>>>>>>>>>>> Yes, we will provide a default
>>> InMemoryCatalogStoreFactory to
>>>>>>> create
>>>>>>>> an
>>>>>>>>>>> InMemoryCatalogStore.
>>>>>>>>>>>
>>>>>>>>>>>> If we get a Catalog from CatalogStore, after
>>> initializing
>>>>> it,
>>>>>>>> whether
>>>>>>>>>> we
>>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
>>>>>>>>>>>
>>>>>>>>>>> Yes, in the current design, catalogs are stored as
>>> snapshots,
>>>>> and
>>>>>>>> once
>>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
>>>>>> Catalog>
>>>>>>>>>>> catalogs.
>>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
>>> the
>>>>>> primary
>>>>>>>>>> source
>>>>>>>>>>> for obtaining the corresponding Catalog.
>>>>>>>>>>>
>>>>>>>>>>>> how about renaming them to `catalog.store.type` and
>>>>>>>>>>> `catalog.store.path`?
>>>>>>>>>>>
>>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
>>> seem a
>>>>> bit
>>>>>>>>>> strange. I
>>>>>>>>>>> will update the FLIP.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> @Shammon
>>>>>>>>>>>
>>>>>>>>>>> Thank you for the review. I have made the necessary
>>>>> corrections.
>>>>>>>>>>> Regarding the modifications made to the Public Interface,
>>> I
>>>>> have
>>>>>>> also
>>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Feng
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
>>>> zjureel@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi feng,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for updating, I have some minor comments
>>>>>>>>>>>>
>>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
>>>>>> `Public
>>>>>>>>>>>> Interfaces`, it is not a public interface.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
>>> and
>>>>>>>>>>>> `CatalogStoreFactory`
>>>>>>>>>>>>
>>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
>>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
>>>>> should
>>>>>> be
>>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
>>>>>>>>>>>> catalogStore.get(catalogName);`
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Shammon FY
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
>>>> ron9.liu@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, Feng
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
>>>> useful
>>>>>> for
>>>>>>>>>> catalog
>>>>>>>>>>>>> management.
>>>>>>>>>>>>> I have some small questions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Regarding the
>>> CatalogStoreFactory#createCatalogStore
>>>>>> method,
>>>>>>>> do we
>>>>>>>>>>>> need
>>>>>>>>>>>>> to provide a default implementation?
>>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
>>>> initializing
>>>>>> it,
>>>>>>>>>> whether
>>>>>>>>>>> we
>>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
>>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
>>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
>>> them to
>>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>
>>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
>>> 21:19写道：
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi yuxia
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
>>>>> register
>>>>>>> the
>>>>>>>>>>> Catalog,
>>>>>>>>>>>>> we
>>>>>>>>>>>>>> initialize it and open it. right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
>>>>>> original
>>>>>>>>>> CREATE
>>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
>>>>>>>>>> registerCatalog
>>>>>>>>>>> so
>>>>>>>>>>>>>> that parameter validation can be performed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In the current design, lazy initialization is mainly
>>>>>>> reflected
>>>>>>>> in
>>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
>>>>> catalog
>>>>>>>>>>>>> configurations,
>>>>>>>>>>>>>> only initialization is required in getCatalog.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
>>>>>>>> luoyuxia@alumni.sjtu.edu.cn
>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, Feng.
>>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
>>>>>>>> initialization*.
>>>>>>>>>>> If
>>>>>>>>>>>>> i'm
>>>>>>>>>>>>>>> wrong, please correct me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
>>>> access
>>>>>> the
>>>>>>>>>>> catalog,
>>>>>>>>>>>>> then
>>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
>>>>> Changes,
>>>>>>>> once we
>>>>>>>>>>>>>> register
>>>>>>>>>>>>>>> the Catalog,
>>>>>>>>>>>>>>> we initialize it and open it. right?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Yuxia
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- 原始邮件 -----
>>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
>>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
>>>>>> Catalog
>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>> and asynchronous registration
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One of the major changes is that current design
>>> will
>>>>>>> provide
>>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
>>>> of a
>>>>>>>> cache,
>>>>>>>>>>> which
>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>> once it has been initialized, any changes done by
>>>> other
>>>>>>>> sessions
>>>>>>>>>>> will
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
>>>>>> further
>>>>>>>>>>>> improvement.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
>>>>>>>> jinfeng1995@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
>>>>>> progress
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
>>> Shammon
>>>> FY,
>>>>>>>> ShengKai
>>>>>>>>>>> Fang
>>>>>>>>>>>>>> and I
>>>>>>>>>>>>>>>> had an offline discussion regarding the overall
>>>>>> solution
>>>>>>>> for
>>>>>>>>>>> Flink
>>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
>>>> have
>>>>>>>> updated
>>>>>>>>>> the
>>>>>>>>>>>>> final
>>>>>>>>>>>>>>>> solution in FLIP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
>>>>>>>> configuration
>>>>>>>>>>>> similar
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> TableDescriptor.
>>>>>>>>>>>>>>>> 2.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
>>>>>>>>>>> storeCatalog(String
>>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
>>>>>> CatalogDescriptor
>>>>>>>>>>>>>>> getCatalog(String)
>>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
>>>> Catalog
>>>>>>>> instance.
>>>>>>>>>>>> This
>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>>>> CatalogStore will only be responsible for
>>> saving
>>>>> and
>>>>>>>>>>> retrieving
>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>> configurations without having to initialize
>>>>>> catalogs.
>>>>>>>>>>>>>>>> 3.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The default registerCatalog(String
>>> catalogName,
>>>>>>> Catalog
>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>> function in CatalogManager will be marked as
>>>>>>> deprecated.
>>>>>>>>>>>>>>>> 4.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> A new function registerCatalog(String
>>>> catalogName,
>>>>>>>>>>>>> CatalogDescriptor
>>>>>>>>>>>>>>>> catalog) will be added to serve as the
>>> default
>>>>>>>> registration
>>>>>>>>>>>>> function
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> catalogs in CatalogManager.
>>>>>>>>>>>>>>>> 5.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
>>>> will
>>>>>>>> remain
>>>>>>>>>>>>> unchanged
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> save initialized catalogs.This means that
>>>> deletion
>>>>>>>>>> operations
>>>>>>>>>>>> from
>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>> session won't synchronize with other
>>> sessions.
>>>>>>>>>>>>>>>> 6.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To support multi-session synchronization
>>>> scenarios
>>>>>> for
>>>>>>>>>>> deletions
>>>>>>>>>>>>>> later
>>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
>>>>>>>>>>> configurable.There
>>>>>>>>>>>>> may
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> three possible situations:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
>>>>>>>> automatically
>>>>>>>>>>> clear
>>>>>>>>>>>>>>>> long-unused catalogs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> c.No caching of any instances; each call to
>>>>>> getCatalog
>>>>>>>>>>> creates a
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>> instance.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is the document for discussion:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is the final proposal document:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you very much for your attention and
>>>>> suggestions
>>>>>> on
>>>>>>>> this
>>>>>>>>>>>>> FLIP. A
>>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
>>> on
>>>>> the
>>>>>>>> entire
>>>>>>>>>>>> design
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> organizing offline discussions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If you have any further suggestions or feedback
>>>> about
>>>>>>> this
>>>>>>>> FLIP
>>>>>>>>>>>>> please
>>>>>>>>>>>>>>> feel
>>>>>>>>>>>>>>>> free to share.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
>>>>>>>>>>> <jing@ververica.com.invalid
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
>>> to
>>>> me.
>>>>>> We
>>>>>>>> could
>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>> reconsider in the future how to provide more
>>>> common
>>>>>>>> built-in
>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
>>> every
>>>>>>>>>> CatalogSotre
>>>>>>>>>>>>>>>>> implementation has to take care of it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
>>>>>>>>>> jinfeng1995@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Jing,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for your reply.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There might be more such issues. I would
>>>>> suggest
>>>>>>> you
>>>>>>>>>>>>> completely
>>>>>>>>>>>>>>> walk
>>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
>>> the
>>>>> same
>>>>>>>> time, I
>>>>>>>>>>>>> greatly
>>>>>>>>>>>>>>>>>> appreciate your careful review. I have
>>>> thoroughly
>>>>>>>> checked
>>>>>>>>>> the
>>>>>>>>>>>>>> entire
>>>>>>>>>>>>>>>> FLIP
>>>>>>>>>>>>>>>>>> and made corrections to these issues.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
>>> FLIP
>>>>>>> design,
>>>>>>>>>>>>>>> CatalogManager
>>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
>>> being
>>>>>>>> configured.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
>>> was
>>>> not
>>>>>>>> necessary
>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
>>>> catalogs
>>>>>> to
>>>>>>>> store
>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>> instances.
>>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
>>>>>>> Therefore, I
>>>>>>>>>>>> modified
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>>> of the design and removed Map<String,
>>> Catalog>
>>>>>>> catalogs
>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> CatalogManager.
>>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
>>> serve
>>>> as
>>>>>> the
>>>>>>>>>> default
>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
>>>>>>>> functionality of
>>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
>>>>>>>>>>>>>>>>>> The previous plan that kept
>>>>>>>> Map<String,Catalog>catalogs has
>>>>>>>>>>>> been
>>>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> Rejected Alternatives.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
>>>>>>>>>>>>> <jing@ververica.com.invalid
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There are still many places contain
>>>>> inconsistent
>>>>>>>> content,
>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
>>> used.
>>>>>>>>>>>>>>>>>>> 2. The java comment of the method
>>>>>>>> registerCatalog(String
>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
>>> not
>>>>> tell
>>>>>>>> what the
>>>>>>>>>>>>> method
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> doing.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There might be more such issues. I would
>>>>> suggest
>>>>>>> you
>>>>>>>>>>>> completely
>>>>>>>>>>>>>>> walk
>>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
>>> issues.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
>>> mean
>>>>> that
>>>>>>> you
>>>>>>>> will
>>>>>>>>>>>> build
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
>>> is a
>>>>> very
>>>>>>>>>> different
>>>>>>>>>>>>>> design
>>>>>>>>>>>>>>>>>> concept
>>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
>>> I am
>>>>> not
>>>>>>>>>> mistaken,
>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
>>>> work
>>>>>>>> without
>>>>>>>>>>>> Optional
>>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
>>>>> reason
>>>>>>>> why I
>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> last email that the example code wrt the
>>>>> Optional
>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> correct.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
>>>>>>>>>>>>> jinfeng1995@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Jing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
>>>> FLIP
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
>>>> review. I
>>>>>>> have
>>>>>>>>>>> already
>>>>>>>>>>>>> made
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> relevant changes.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
>>> the
>>>>>>>>>> multi-instance
>>>>>>>>>>>>>>>> scenarios, I
>>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
>>> multiple
>>>>>>>> scenarios
>>>>>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
>>>>> remove
>>>>>>>>>>> 'Map<String,
>>>>>>>>>>>>>>>> Catalog>'
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
>>>>>>>>>> InMemoryCatalogStore.
>>>>>>>>>>>> This
>>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
>>>>> achieve
>>>>>>>>>>>>> multi-instance
>>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
>>>> think?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
>>>>>>>>>>>>>>> <jing@ververica.com.invalid
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
>>> are
>>>>>> still
>>>>>>>> some
>>>>>>>>>>> NIT
>>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> FLIP
>>>>>>>>>>>>>>>>>>>>> like:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
>>>> has
>>>>>> been
>>>>>>>> used
>>>>>>>>>> as
>>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
>>> example.
>>>> It
>>>>>>>> should be
>>>>>>>>>>>> fine
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
>>> after
>>>> you
>>>>>>>> submit
>>>>>>>>>> the
>>>>>>>>>>>> PR.
>>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
>>>> somewhere
>>>>> in
>>>>>>> the
>>>>>>>>>>>> rejected
>>>>>>>>>>>>>>>> section
>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
>>> it
>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
>>>>>>>> multi-instance
>>>>>>>>>>>>>>> scenarios,
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> sure if this is the intended
>>> behaviour.
>>>> If
>>>>>>>>>> Map<String,
>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>> catalogs
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
>>>> once
>>>>>> the
>>>>>>>>>> related
>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
>>> another
>>>>>>>> instance.
>>>>>>>>>> Did
>>>>>>>>>>> I
>>>>>>>>>>>>> miss
>>>>>>>>>>>>>>>>>>> something?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
>>> Jin
>>>> <
>>>>>>>>>>>>>>> jinfeng1995@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
>>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> @Jing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
>>>>>>>>>>>>>>>>>>>>>> I think this is a good function
>>> name, I
>>>>>> have
>>>>>>>>>> updated
>>>>>>>>>>> it
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Some common cache features
>>> should
>>>> be
>>>>>>>>>> implemented
>>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
>>>>>> alternative
>>>>>>> 1
>>>>>>>>>> turns
>>>>>>>>>>>> out
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
>>> this
>>>>> part
>>>>>>> of
>>>>>>>> the
>>>>>>>>>>>>> content.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
>>>> forward,
>>>>>> the
>>>>>>>>>> option 2
>>>>>>>>>>>>>>> solution
>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
>>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
>>>>> offline,
>>>>>>> we
>>>>>>>>>> think
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> solution
>>>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
>>>>>>>> inconsistency
>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
>>>>> descriptions
>>>>>> in
>>>>>>>> the
>>>>>>>>>>>>> content.
>>>>>>>>>>>>>>>> Would
>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>> to clean them up?
>>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
>>>> document
>>>>>> and
>>>>>>>>>>>> appreciate
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> @Shammon
>>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
>>> in
>>>>>>>> `Rejected
>>>>>>>>>>>>>>> Alternatives`
>>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
>>> `Rejected
>>>>>>>>>> Alternatives`.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
>>> Shammon
>>>>> FY
>>>>>> <
>>>>>>>>>>>>>>> zjureel@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
>>>>>> `Proposed
>>>>>>>>>>> Changes`,
>>>>>>>>>>>>> can
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> put
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
>>>>>>>> Alternatives`? I
>>>>>>>>>>> think
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>>> us
>>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
>>> Jing
>>>> Ge
>>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
>>> am
>>>>>> still
>>>>>>>>>> confused
>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>> completely
>>>>>>>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>>>>>>>> the thread with following
>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
>>>>> registerCatalog()
>>>>>>> and
>>>>>>>>>>>>>> addCatalog()
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>> big
>>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
>>>> One
>>>>> of
>>>>>>>> them is
>>>>>>>>>>>>>>> responsible
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
>>>>>> persistCatalog()?
>>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
>>>> Map<String,
>>>>>>>> Catalog>
>>>>>>>>>>>>> catalogs
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
>>> data
>>>>>>>> persistence.
>>>>>>>>>> I
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>> describing
>>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
>>>>> clearly
>>>>>> in
>>>>>>>> the
>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>> Some
>>>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
>>>> i.e.
>>>>>> data
>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
>>>> instance
>>>>>>>> should be
>>>>>>>>>>>> found
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
>>>>>> initialized
>>>>>>>> or it
>>>>>>>>>>> will
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> lazy
>>>>>>>>>>>>>>>>>>>>>> initialized)
>>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
>>>>>>> consistency
>>>>>>>> will
>>>>>>>>>>> be
>>>>>>>>>>>>>> taken
>>>>>>>>>>>>>>>> care
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> while
>>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
>>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
>>>>>> forward,
>>>>>>>> the
>>>>>>>>>>>> option
>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>> solution
>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
>>>>>> because,
>>>>>>>> afaiu,
>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
>>>>> solved
>>>>>>>> yet. Do
>>>>>>>>>>> you
>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> propose
>>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
>>>> suggestions
>>>>>> for
>>>>>>>> both
>>>>>>>>>> of
>>>>>>>>>>>>> them?
>>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
>>>> there
>>>>>> are
>>>>>>>> some
>>>>>>>>>>>>>>> inconsistent
>>>>>>>>>>>>>>>>>>>>> descriptions
>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
>>>> clean
>>>>>>> them
>>>>>>>> up?
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
>>> Feng
>>>>>> Jin <
>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
>>> and
>>>> I
>>>>>>>> completely
>>>>>>>>>>>> agree
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> view.
>>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
>>>>>> complicated
>>>>>>>> the
>>>>>>>>>>> whole
>>>>>>>>>>>>>> issue.
>>>>>>>>>>>>>>>>> First
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> foremost,
>>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
>>>> persistence
>>>>>> of
>>>>>>>> the
>>>>>>>>>>>>> Catalog's
>>>>>>>>>>>>>>>>>>>> Configuration.
>>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
>>>>>> persistence
>>>>>>>> for
>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>> Configuration,
>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
>>> to
>>>> the
>>>>>>>> Catalog
>>>>>>>>>>>>>> interface.
>>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
>>> Catalog
>>>>>>>> instance
>>>>>>>>>> to a
>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>> String>
>>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
>>>>>>>> implementation
>>>>>>>>>> will
>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
>>>>>>>>>>>>>>>>>>>>>>>>> /**
>>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
>>> the
>>>>>>>> properties
>>>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>>>> object.
>>>>>>>>>>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
>>> the
>>>>>>>> properties
>>>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>>>> object
>>>>>>>>>>>>>>>>>>>>>>>>> * @throws
>>>>>>>> UnsupportedOperationException if
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> implementing
>>>>>>>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>>>>>>>>>>> not override
>>>>>>>>>>>>>>>>>>>>>>>>> * the default
>>> implementation
>>>> of
>>>>>> this
>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> */
>>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
>>>>>>>> toProperties()
>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>>>>>>>> throw new
>>>>>>>>>>>> UnsupportedOperationException("Please
>>>>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>>>>>>>>> toProperties
>>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
>>>> follows:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
>>> configured a
>>>>>>>>>> CatalogStore,
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> toConfiguration()
>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> will be called when
>>> registering a
>>>>>>> Catalog
>>>>>>>>>>>> instance
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>>>> catalogName,
>>>>>>>> Catalog
>>>>>>>>>>>>> catalog).
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
>>>> Map<String,
>>>>>>>> String>
>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> saved
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
>>>>>> instances
>>>>>>>> do
>>>>>>>>>> not
>>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> method,
>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
>>>>>>>> CatalogStore,
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> toConfiguration()
>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
>>>>>>> consistency
>>>>>>>> with
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> original
>>>>>>>>>>>>>>>>>>>> process.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
>>>>>> Catalog>
>>>>>>>>>>> catalogs
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
>>>>>> conflicts
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
>>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
>>>>> itself
>>>>>>> and
>>>>>>>>>>> provide
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> dedicated
>>>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
>>> loading,
>>>>> such
>>>>>>> as
>>>>>>>>>>>>>>>>> ConfigurationCatalog
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> public class
>>> ConfigurationCatalog
>>>>>>>> implements
>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>>>>>>>>
>>> ConfigurationCatalog(Map<String,
>>>>>>> String>
>>>>>>>>>>>>> properties)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
>>> the
>>>>> FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
>>> 10:03 AM
>>>>>> Shammon
>>>>>>>> FY <
>>>>>>>>>>>>>>>>>> zjureel@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
>>> the
>>>>>>>> functionality
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> `CatalogStore`,
>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
>>>> multiple
>>>>>>>>>>>>> `registerCatalog`
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
>>>>>> implementation
>>>>>>> in
>>>>>>>>>> Trino
>>>>>>>>>>>> is
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> same,
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
>>> only
>>>>> has
>>>>>>> one
>>>>>>>>>>>>>>> `createCatalog`,
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
>>>>>> `CatalogStore`.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
>>>>>> another
>>>>>>>>>>>>>>> `registerCatalog`
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
>>> `Map<String,
>>>>>>> Catalog>
>>>>>>>>>>>> catalogs`
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> `CatalogStore
>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
>>>> mentioned
>>>>>>>> before,
>>>>>>>>>> the
>>>>>>>>>>>>>>>> `Map<String,
>>>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
>>>>>>>> `CatalogManager`.
>>>>>>>>>> How
>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>> saving
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
>>> catalogs`
>>>>> and
>>>>>>>>>>>> `CatalogStore
>>>>>>>>>>>>>>>>>>> catalogStore`
>>>>>>>>>>>>>>>>>>>>>>>> together
>>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
>>> added
>>>> in
>>>>>>>>>>>>> `CatalogManager`?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>>> 5:22 PM
>>>>> Feng
>>>>>>>> Jin <
>>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
>>> I
>>>> am
>>>>>> very
>>>>>>>> sorry
>>>>>>>>>>> for
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> misunderstanding
>>>>>>>>>>>>>>>>>>>>>>>>> caused
>>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
>>>>> original
>>>>>>>>>>> discussion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
>>>>>> pre-discussion
>>>>>>>> [1]
>>>>>>>>>> for
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> FLIP
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
>>> such a
>>>>>>>> discussion
>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> designing
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
>>> that
>>>>> the
>>>>>>>> logic of
>>>>>>>>>>>>>>>>> CatalogManager
>>>>>>>>>>>>>>>>>>>> itself
>>>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
>>>> cannot
>>>>>>> only
>>>>>>>>>>> persist
>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
>>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
>>>>>>>> registerCatalog(String
>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
>>>> Catalogs,
>>>>>>>> because
>>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
>>>>> corresponding
>>>>>> to
>>>>>>>> the
>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>> instances.
>>>>>>>>>>>>>>>>>>>>>>>> Therefore,
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
>>>>>>> CatalogStore
>>>>>>>>>>>> interface
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
>>> this
>>>>>> part, I
>>>>>>>> also
>>>>>>>>>>> took
>>>>>>>>>>>>>>>>> inspiration
>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>> Trino's
>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
>>>> handling
>>>>>>>> catalog
>>>>>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>> looks a
>>>>>>>>>>>>>>>>>>>>>>>>> little
>>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
>>>> suggestion. I
>>>>>>> will
>>>>>>>>>>> provide
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> detailed
>>>>>>>>>>>>>>>>>>>>>> description
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
>>> part,
>>>>> and
>>>>>>> add
>>>>>>>> it
>>>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
>>>>>>>> `addCatalog`
>>>>>>>>>>> still
>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>> confused
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
>>>>>> catalog
>>>>>>> to
>>>>>>>>>>>>>> CatalogStore,
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
>>> very
>>>>>>> personal
>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> decide
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
>>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
>>>>> intuitive
>>>>>>> to
>>>>>>>> me.
>>>>>>>>>> I
>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
>>>>> statement,
>>>>>>>> which
>>>>>>>>>>>> method
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> called?
>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
>>>>>> `addCatalog`?
>>>>>>>> I'm
>>>>>>>>>>>>> wondering
>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
>>>>>> stement.
>>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
>>>> original
>>>>>>>> design
>>>>>>>>>> was
>>>>>>>>>>> to
>>>>>>>>>>>>> add
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> directly
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
>>> would
>>>>>>> disrupt
>>>>>>>> the
>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>> logic.
>>>>>>>>>>>>>>>>>>>>>>> Therefore, I
>>>>>>>>>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
>>>>>>>> configuration to
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> initialize
>>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
>>> the
>>>>> same
>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
>>> neccessary
>>>> to
>>>>>>>> provide a
>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>> implmentation
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
>>>>> `CatalogStoreFactory`?
>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
>>>>>> otherwise
>>>>>>> we
>>>>>>>>>> would
>>>>>>>>>>>>> need
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> introduce
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>> additional
>>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
>>>> configuration
>>>>>> for
>>>>>>>> lazy
>>>>>>>>>>>>> loading.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
>>>>>>> registration
>>>>>>>> for
>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
>>>>>>> registerCatalog(String
>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
>>>>>> interface
>>>>>>>>>> because
>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>> instance.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
>>>>>>>> initialization
>>>>>>>>>>> for
>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
>>>>>>>> inaccurate. It
>>>>>>>>>>>>> should
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> lazy
>>>>>>>>>>>>>>>>>>>>>>> registration
>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
>>>>>>>> registration. I
>>>>>>>>>>> have
>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> title
>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>>>> 4:27 PM
>>>>>>>> Shammon
>>>>>>>>>> FY <
>>>>>>>>>>>>>>>>>>> zjureel@gmail.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
>>>>>>>> `registerCatalog`
>>>>>>>>>>>>> method
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
>>>>> method
>>>>>>>> named
>>>>>>>>>>>>>> `addCatalog`
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
>>> the
>>>>>> memory
>>>>>>>>>> catalog
>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>> `cache`,
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> concept
>>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
>>>>>> exposed
>>>>>>> to
>>>>>>>>>>> users.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
>>>>>> pre-discussion
>>>>>>>> [1]
>>>>>>>>>> for
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>>>> Please
>>>>>>>>>>>>>>>>>>>>>>> correct
>>>>>>>>>>>>>>>>>>>>>>>>> me
>>>>>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
>>>>> conclusion
>>>>>>> of
>>>>>>>> that
>>>>>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
>>>>>> interface
>>>>>>>> and
>>>>>>>>>>>>> implement
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>>>>>>>>>>> stores
>>>>>>>>>>>>>>>>>>>>>>>>>>> such
>>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
>>>> external
>>>>>>>> system.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
>>>>> between
>>>>>>> the
>>>>>>>>>>> current
>>>>>>>>>>>>> FLIP
>>>>>>>>>>>>>>>>> design
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
>>> of
>>>>> the
>>>>>>>>>> discussion
>>>>>>>>>>>> in
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>>> [1] ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>>>>> 3:41 PM
>>>>>>>> yuxia <
>>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
>>> driving
>>>>> this
>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
>>>>> handling
>>>>>>>> catalog
>>>>>>>>>>> with
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>> looks a
>>>>>>>>>>>>>>>>>>>>>>>>>> little
>>>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
>>>>> it'll
>>>>>> be
>>>>>>>>>> better
>>>>>>>>>>> to
>>>>>>>>>>>>>>> explain
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> java
>>>>>>>>>>>>>>>>>>>>>>>>> doc
>>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
>>> give a
>>>>>> brief
>>>>>>>>>> example
>>>>>>>>>>> in
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
>>> name
>>>>>>>> `addCatalog`
>>>>>>>>>>>> still
>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>> confused
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
>>>> storing
>>>>>>>> catalog to
>>>>>>>>>>>>>>>> CatalogStore,
>>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
>>> It's
>>>>> very
>>>>>>>> personal
>>>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> decide
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
>>>>>>> statement,
>>>>>>>>>> which
>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> called?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
>>>>>>>> `addCatalog`?
>>>>>>>>>> I'm
>>>>>>>>>>>>>>> wondering
>>>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
>>> SQL
>>>>>>>> stement.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
>>>> neccessary
>>>>> to
>>>>>>>>>> provide a
>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>> implmentation
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
>>>>>>> `CatalogStoreFactory`?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
>>>>>>>> registration
>>>>>>>>>> for
>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
>>> catalog
>>>>>> with
>>>>>>>> CREATE
>>>>>>>>>>>>>> CATALOG,
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
>>>> used
>>>>>> by
>>>>>>>>>> default.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
>>>>> registration
>>>>>> is
>>>>>>>> the
>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>> behavior,
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
>>> to
>>>>>>>> synchronous
>>>>>>>>>>>>>>> registration
>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>> before?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
>>>>> `addCatalog`
>>>>>>> and
>>>>>>>>>>>>>>>> `registerCatalog`
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
>>>>>>>> registration,
>>>>>>>>>> it
>>>>>>>>>>>> may
>>>>>>>>>>>>>> well
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> CREATE
>>>>>>>>>>>>>>>>>>>>>>>> CATALOG
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
>>>> but
>>>>>> then
>>>>>>>> the
>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>> CREATE
>>>>>>>>>>>>>>>>>>>> TABLE
>>>>>>>>>>>>>>>>>>>>>>>>> statement
>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
>>> fail
>>>>> to
>>>>>>>> open.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
>>>> change
>>>>>>> which
>>>>>>>>>>> should
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> highlighted
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
>>>>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
>>>> asynchronous
>>>>>>>>>>>> registration, I
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> expect
>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
>>>> open
>>>>>> or
>>>>>>>>>> register
>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> background,
>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
>>> comments,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
>>>> will
>>>>> be
>>>>>>>>>>> initialized
>>>>>>>>>>>>> if
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
>>>> initialization
>>>>>>>> process
>>>>>>>>>>> fails,
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>> statements
>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
>>> successfully."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
>>> lazy
>>>>>>>>>> initialization
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
>>>>>>>>>> jinfeng1995@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
>>>>>>> dev@flink.apache.org
>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
>>> 03日
>>>> 下午
>>>>>>>> 3:27:45
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
>>>> 295:
>>>>>>>> Support
>>>>>>>>>>>>>> persistence
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
>>>>> registration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
>>>> all
>>>>>> for
>>>>>>>> your
>>>>>>>>>>>>> interest
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> DISCUSS.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
>>> catalog
>>>>>> with
>>>>>>>> the
>>>>>>>>>> same
>>>>>>>>>>>>> name
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> exists
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
>>>> crucial
>>>>>>>> point.
>>>>>>>>>>> Based
>>>>>>>>>>>> on
>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>>>>> personal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
>>>>>> Map<String,
>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>> catalogs
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> serve
>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
>>> catalogs
>>>>> and
>>>>>>>> have
>>>>>>>>>> the
>>>>>>>>>>>>>> highest
>>>>>>>>>>>>>>>>>>> priority.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
>>> methods
>>>>> that
>>>>>>> can
>>>>>>>>>> affect
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
>>> registerCatalog(String
>>>>>>>>>> catalogName,
>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
>>>>> catalog
>>>>>>>> instance
>>>>>>>>>>>> into
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
>>>> unregisterCatalog(String
>>>>>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
>>> the
>>>>>> catalog
>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>> corresponding
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> catalogName
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
>>>>> Catalog>
>>>>>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
>>>> retrieves
>>>>>> the
>>>>>>>>>>>>> corresponding
>>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
>>> Catalog>
>>>>>>>> catalogs. If
>>>>>>>>>>> the
>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> exist,
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
>>>> corresponding
>>>>>>>>>>> configuration
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
>>> puts
>>>>> the
>>>>>>>>>>> initialized
>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
>>> methods
>>>>>> only
>>>>>>>> modify
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
>>>>>>> catalogName,
>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>> String>
>>>>>>>>>>>>>>>>>>>>>>> properties)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
>>>>>>> properties
>>>>>>>> to
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> catalogStore
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
>>>>>> catalogName
>>>>>>>> with
>>>>>>>>>> the
>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
>>> removeCatalog(String
>>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
>>> the
>>>>>>> specified
>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> specified
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
>>>>>>> catalogStore.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
>>>> possible
>>>>>>>> conflict
>>>>>>>>>>>>>> scenarios:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
>>> corresponding
>>>>>>>> catalogName
>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> exists
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
>>> in
>>>> the
>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>> Catalog>,
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>>>>>>>> catalogName,
>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
>>> directly
>>>>> saved
>>>>>>> to
>>>>>>>> the
>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
>>> corresponding
>>>>>>>> catalogName
>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> exists
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
>>>>>>> Map<String,
>>>>>>>>>>>> Catalog>,
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
>>>>> catalog)
>>>>>>>> method
>>>>>>>>>>> will
>>>>>>>>>>>>>> fail.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
>>> corresponding
>>>>>>>> catalogName
>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> exists
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
>>> the
>>>>>>>>>>>> addCatalog(String
>>>>>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
>>>> method
>>>>>> can
>>>>>>>>>> directly
>>>>>>>>>>>>> save
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
>>>>>>>>>> getCatalog(String
>>>>>>>>>>>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
>>>> for
>>>>>>>>>>> initialization
>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> corresponding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
>>> already
>>>>>> exists
>>>>>>>> in
>>>>>>>>>>>> catalogs
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
>>>>>>>>>>>>> unregisterCatalog(String
>>>>>>>>>>>>>>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
>>>>>>>> corresponding to
>>>>>>>>>>> the
>>>>>>>>>>>>>>> original
>>>>>>>>>>>>>>>>>>>>> catalogName
>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
>>> confuse
>>>>>> users
>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
>>>>> catalog)`
>>>>>>> in
>>>>>>>> the
>>>>>>>>>>>>>>> `Map<String,
>>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>> `registerCatalog(String
>>>>>>>> catalogName,
>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>> String>
>>>>>>>>>>>>>>>>>>>>>>>>> properties)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
>>>> lead
>>>>>> to
>>>>>>>>>>>> confusion. I
>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>> changing
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
>>> to
>>>>>>>>>>> addCatalog(String
>>>>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
>>> as
>>>>>>>> previously
>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
>>>>> `registerCatalog(String
>>>>>>>>>>>>>>> catalogName,Catalog
>>>>>>>>>>>>>>>>>>>> catalog,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
>>>> already
>>>>> an
>>>>>>>>>> instance,
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> "lazyInit"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
>>>>>>>>>> registerCatalog(String
>>>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
>>>> necessarily
>>>>>>>> result in
>>>>>>>>>>> lazy
>>>>>>>>>>>>>>>>>>> initialization.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
>>>> about
>>>>>>>>>> encryption
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
>>>>>>> necessary,
>>>>>>>> but
>>>>>>>>>>>>> perhaps
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> encryption
>>>>>>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
>>> in
>>>>> the
>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
>>> the
>>>>>>>>>>> FileCatalogStore
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> encrypt
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
>>>>> while
>>>>>>> the
>>>>>>>>>>>>>>>> MemoryCatalogStore
>>>>>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>> require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
>>> the
>>>>>>>>>>>>>> `MemoryCatalogStore`?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
>>> necessary
>>>> to
>>>>>>> have a
>>>>>>>>>>>>>>>>> MemoryCatalogStore
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
>>>> saves
>>>>>>>> Catalog
>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
>>> to
>>>>>>>> implement
>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>> loading
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>> Catalog,
>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
>>> introduce
>>>>>>>> additional
>>>>>>>>>>> cache.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
>>>>>>>> registration
>>>>>>>>>>>> failed?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
>>> critical
>>>>>>> concern.
>>>>>>>>>> When
>>>>>>>>>>>>>>> executing
>>>>>>>>>>>>>>>>> DDL,
>>>>>>>>>>>>>>>>>>>> DQL,
>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> DML
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
>>>> reference a
>>>>>>>> specific
>>>>>>>>>>>>> Catalog,
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
>>> if it
>>>>> has
>>>>>>> not
>>>>>>>>>> been
>>>>>>>>>>>>>>>> initialized
>>>>>>>>>>>>>>>>>> yet.
>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
>>>>> fails,
>>>>>>>> these
>>>>>>>>>>>>>> statements
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> executed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
>>>>>> Catalog>
>>>>>>>>>> catalogs
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
>>>> have
>>>>>> the
>>>>>>>> same
>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>> name,
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>> correponding
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
>>>>> instances.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
>>>> resolved
>>>>> by
>>>>>>>>>> referring
>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>>>>> responses.
>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
>>> the
>>>>>>> Catalog
>>>>>>>>>> that
>>>>>>>>>>>> has
>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>> recently
>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
>>>> for
>>>>>>>> subsequent
>>>>>>>>>>>> use.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
>>>>>> 10:58 PM
>>>>>>>>>> Xianxun
>>>>>>>>>>>> Ye
>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
>>> this
>>>>>> Flip,
>>>>>>>> I do
>>>>>>>>>>>>> believe
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> Flip
>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>> helpful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
>>>> work
>>>>>> for
>>>>>>>> also
>>>>>>>>>>>>> manages a
>>>>>>>>>>>>>>> lot
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> catalogs,
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
>>>> becomes
>>>>>>> slow
>>>>>>>>>>> because
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> loading a
>>>>>>>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
>>> catalogs in
>>>>> the
>>>>>>>> user's
>>>>>>>>>>>> Flink
>>>>>>>>>>>>>> SQL
>>>>>>>>>>>>>>>>>> through
>>>>>>>>>>>>>>>>>>>>>> regular
>>>>>>>>>>>>>>>>>>>>>>>>>>> expression
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
>>>>> catalogs
>>>>>>> to
>>>>>>>>>>> improve
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> speed
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>>>> submission.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
>>> flip-295, I
>>>>>> have
>>>>>>>> some
>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
>>>>> catalog
>>>>>>> with
>>>>>>>>>>> CREATE
>>>>>>>>>>>>>>> CATALOG,
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
>>>> used
>>>>>> by
>>>>>>>>>> default.
>>>>>>>>>>>>> What
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>> registration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
>>>>> implement
>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>> registration?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
>>> same
>>>>>>>> question
>>>>>>>>>> with
>>>>>>>>>>>>>>> Shammon’s
>>>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>>>>>> comment.
>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
>>>>> catalogs
>>>>>>> and
>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>> catalogstore
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
>>> have
>>>> the
>>>>>>> same
>>>>>>>>>>> catalog
>>>>>>>>>>>>>> name,
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>> correponding
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
>>>>> instances.
>>>>>>> For
>>>>>>>>>>> example:
>>>>>>>>>>>>>>>>> catalog1->
>>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
>>> catalog1 ->
>>>>>>>> hiveCatalog
>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> catalogstore.
>>>>>>>>>>>>>>>>>>>>>>> Would
>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
>>> should
>>>>>> define
>>>>>>>>>>> `catalogs`
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> `catalogstore`
>>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
>>> 11:13，Hang
>>>>>> Ruan
>>>>>>> <
>>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 写道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi everyone,
Thank you everyone for your valuable input. If there are no further
questions or concerns about the FLIP[1], I would like to start voting
tomorrow (6/7).
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
Best,
Feng
On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com> wrote:
> Hi Samrat,
>
> Thanks for your advice.
>
> > 1. The createCatalog method does not mention any exceptions being
> thrown.
>
> CreateCatalog will throw CatalogException like registerCatalog. As
> CatalogException is a RuntimeException,
> there is no explicit declaration of throwing Exceptions in CatalogManager
> and TableEnvironment.
> To avoid misunderstandings, I have added the "throw CatalogException" flag
> to the createCatalog method definition of CatalogStore.
>
> > 2. Could you please provide an exhaustive list of the supported kinds?
>
> Sure, the documentation now includes the configuration of the built-in
> CatalogStore as well as how to configure a custom CatalogStore.
>
>
> Best,
> Feng
>
>
> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com> wrote:
>
>> Hi Feng,
>>
>> Thank you for providing the proposal. I believe this feature will be
>> highly
>> valuable.
>>
>> I have a couple of inquiries:
>>
>> 1. According to the documentation [1], the createCatalog method does not
>> mention any exceptions being thrown. However, I would like to confirm if
>> it
>> is always true that there will be no failures in all scenarios. Please let
>> me know if there is any additional information I may have missed.
>>
>> 2. Regarding the registration process using the `table.catalog-store.kind`
>> configuration, could you please provide an exhaustive list of the
>> supported
>> kinds?
>> It would be great to have a comprehensive understanding of the options
>> available.
>>
>> Bests,
>> Samrat
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>>
>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
>>
>> > Hi Hang and Jark
>> >
>> > Thank you very much for your reply.
>> >
>> > @Jark
>> > > 1. Could you move the CatalogStore registration API to the "Public
>> > Interface" section?
>> >
>> > Indeed, it is more reasonable.
>> >
>> > > 2. We should prefix "table." for the CatalogStore configuration.
>> >
>> > Sure, the names that you provided are indeed more logical and easier
>> to
>> > understand.
>> >
>> > > 3. About the open/close method in CatalogStoreFactory.
>> >
>> > I also agree with the proposed requirement scenario and design. I have
>> > already made modifications to the interface.
>> >
>> > @Hang
>> >
>> > > 1. The `CatalogStore` need the `open`/`close` methods to init or close
>> > the resource.
>> >
>> > I have already added the missing method.
>> >
>> > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>> >
>> > Sorry for the misspelling, I updated this part.
>> >
>> > > 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
>> > to init or close its resource
>> >
>> > This is indeed a scenario that needs to be considered, and I have added
>> > relevant methods accordingly.
>> > Additionally, I have included some relevant description information in
>> the
>> > documentation to help others understand
>> > why the open/close methods need to be added.
>> >
>> >
>> > Best,
>> > Feng
>> >
>> > On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
>> >
>> > > Hi Jing,
>> > >
>> > > Thank you for the update.
>> > >
>> > > 1. Could you move the CatalogStore registration API to the "Public
>> > > Interface" section?
>> > > "Proposed Changes" is more like a place to describe the implementation
>> > > details.
>> > >
>> > > 2. We should prefix "table." for the CatalogStore configuration.
>> Besides,
>> > > the config key
>> > > name should be hierarchical[1]. Therefore, it may be better to use:
>> > > "table.catalog-store.kind"
>> > > "table.catalog-store.file.path"
>> > >
>> > > 3. I think Hang's suggestions make sense.
>> > >
>> > > Others look good to me.
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > > On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
>> wrote:
>> > >
>> > > > Hi, Feng.
>> > > >
>> > > > Thanks for the update.
>> > > > The current design looks good to me. I have some minor comments.
>> > > >
>> > > > 1. The `CatalogStore` need the `open`/`close` methods to init or
>> close
>> > > the
>> > > > resource. For example, when storing the information in MySQL, the
>> store
>> > > > needs to open and close the connections.
>> > > >
>> > > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>> > > >
>> > > > 3. About the usage in the sql gateway.
>> > > > Considering the usage in sql gateway, the sql gateway may create a
>> > > > CatalogStore for each session.
>> > > > If we are using the MySqlCatalogStore, there would be so many
>> > > connections.
>> > > > How can we reuse the connection among these sessions?
>> > > > I think sql gateway need to maintain a connection pool in
>> > > > the CatalogStoreFactory and each session get its own connection from
>> > the
>> > > > pool when it is created.
>> > > > Then the `CatalogStoreFactory` may need the `open`/`close` methods
>> to
>> > > init
>> > > > or close its resource.
>> > > > Or is there a better way?
>> > > >
>> > > > Best,
>> > > > Hang
>> > > >
>> > > > Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
>> > > >
>> > > > > Thanks Jingsong.
>> > > > >
>> > > > > > Just naming, maybe `createCatalog` in TableEnv
>> > > > >
>> > > > > +1 For this, I have already updated FLIP.
>> > > > >
>> > > > >
>> > > > > Best,
>> > > > > Feng
>> > > > >
>> > > > >
>> > > > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
>> jingsonglee0@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Thanks Feng,
>> > > > > >
>> > > > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
>> > > > > > functions are converted to createxxx from registerxxx.
>> > > > > >
>> > > > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1995@gmail.com
>> >
>> > > > wrote:
>> > > > > > >
>> > > > > > > Hi jark, thanks for your suggestion.
>> > > > > > >
>> > > > > > > > 1. How to register the CatalogStore for Table API? I think
>> the
>> > > > > > > CatalogStore should be immutable once TableEnv is created.
>> > > Otherwise,
>> > > > > > there
>> > > > > > > might be some data inconsistencies when CatalogStore is
>> changed.
>> > > > > > >
>> > > > > > > Yes, We should initialize the CatalogStore when creating the
>> > > > TableEnv.
>> > > > > > > Therefore, my current proposal is to add a method to configure
>> > the
>> > > > > > > CatalogStore in EnvironmentSettings.
>> > > > > > >
>> > > > > > > // Initialize a catalog Store
>> > > > > > > CatalogStore catalogStore = new FileCatalogStore("");
>> > > > > > >
>> > > > > > > // set up the Table API
>> > > > > > > final EnvironmentSettings settings =
>> > > > > > >
>> EnvironmentSettings.newInstance().inBatchMode()
>> > > > > > > .withCatalogStore(catalogStore)
>> > > > > > > .build();
>> > > > > > >
>> > > > > > > final TableEnvironment tableEnv =
>> > > > > > TableEnvironment.create(settings);
>> > > > > > >
>> > > > > > >
>> > > > > > > > 2. Why does the CatalogStoreFactory interface only have a
>> > default
>> > > > > > method,
>> > > > > > > not an interface method?
>> > > > > > >
>> > > > > > > Sorry, While I did refer to the Catalog interface, I agree
>> that
>> > as
>> > > a
>> > > > > new
>> > > > > > > interface, the CatalogStoreFactory should not have a default
>> > method
>> > > > but
>> > > > > > an
>> > > > > > > interface method. I have already modified the interface.
>> > > > > > >
>> > > > > > > > 3. Please mention the alternative API in Javadoc for the
>> > > > > > > deprecated`registerCatalog`.
>> > > > > > > > 4. In the "Compatibility" section, would be better to
>> mention
>> > the
>> > > > > > changed
>> > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
>> > other
>> > > > > > > persisted catalog store) is used.
>> > > > > > >
>> > > > > > > Thanks for the suggestion, I have updated the FLIP [1].
>> > > > > > >
>> > > > > > >
>> > > > > > > [1].
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > > > > >
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Feng
>> > > > > > >
>> > > > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
>> wrote:
>> > > > > > >
>> > > > > > > > Hi Feng,
>> > > > > > > >
>> > > > > > > > This is a useful FLIP. Thanks for starting this discussion.
>> > > > > > > > The current design looks pretty good to me. I just have some
>> > > minor
>> > > > > > > > comments.
>> > > > > > > >
>> > > > > > > > 1. How to register the CatalogStore for Table API? I think
>> the
>> > > > > > CatalogStore
>> > > > > > > > should be immutable once TableEnv is created. Otherwise,
>> there
>> > > > might
>> > > > > be
>> > > > > > > > some data inconsistencies when CatalogStore is changed.
>> > > > > > > >
>> > > > > > > > 2. Why does the CatalogStoreFactory interface only have a
>> > default
>> > > > > > method,
>> > > > > > > > not an interface method?
>> > > > > > > >
>> > > > > > > > 3. Please mention the alternative API in Javadoc for the
>> > > deprecated
>> > > > > > > > `registerCatalog`.
>> > > > > > > >
>> > > > > > > > 4. In the "Compatibility" section, would be better to
>> mention
>> > the
>> > > > > > changed
>> > > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
>> > > other
>> > > > > > > > persisted catalog store) is used.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Jark
>> > > > > > > >
>> > > > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <
>> jinfeng1995@gmail.com>
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi ， thanks all for reviewing the flip.
>> > > > > > > > >
>> > > > > > > > > @Ron
>> > > > > > > > >
>> > > > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore
>> > method,
>> > > > do
>> > > > > we
>> > > > > > > > need
>> > > > > > > > > to provide a default implementation?
>> > > > > > > > >
>> > > > > > > > > Yes, we will provide a default
>> InMemoryCatalogStoreFactory to
>> > > > > create
>> > > > > > an
>> > > > > > > > > InMemoryCatalogStore.
>> > > > > > > > >
>> > > > > > > > > > If we get a Catalog from CatalogStore, after
>> initializing
>> > > it,
>> > > > > > whether
>> > > > > > > > we
>> > > > > > > > > put it in Map<String, Catalog> catalogs again?
>> > > > > > > > >
>> > > > > > > > > Yes, in the current design, catalogs are stored as
>> snapshots,
>> > > and
>> > > > > > once
>> > > > > > > > > initialized, the Catalog will be placed in the Map<String,
>> > > > Catalog>
>> > > > > > > > > catalogs.
>> > > > > > > > > Subsequently, the Map<String, Catalog> catalogs will be
>> the
>> > > > primary
>> > > > > > > > source
>> > > > > > > > > for obtaining the corresponding Catalog.
>> > > > > > > > >
>> > > > > > > > > > how about renaming them to `catalog.store.type` and
>> > > > > > > > > `catalog.store.path`?
>> > > > > > > > >
>> > > > > > > > > I think it is okay. Adding "sql" at the beginning may
>> seem a
>> > > bit
>> > > > > > > > strange. I
>> > > > > > > > > will update the FLIP.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > @Shammon
>> > > > > > > > >
>> > > > > > > > > Thank you for the review. I have made the necessary
>> > > corrections.
>> > > > > > > > > Regarding the modifications made to the Public Interface,
>> I
>> > > have
>> > > > > also
>> > > > > > > > > included the relevant changes to the `TableEnvironment`.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Best,
>> > > > > > > > > Feng
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <
>> > zjureel@gmail.com>
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi feng,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for updating, I have some minor comments
>> > > > > > > > > >
>> > > > > > > > > > 1. The modification of `CatalogManager` should not be in
>> > > > `Public
>> > > > > > > > > > Interfaces`, it is not a public interface.
>> > > > > > > > > >
>> > > > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore`
>> and
>> > > > > > > > > > `CatalogStoreFactory`
>> > > > > > > > > >
>> > > > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
>> > > > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
>> > > should
>> > > > be
>> > > > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
>> > > > > > > > > > catalogStore.get(catalogName);`
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Shammon FY
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <
>> > ron9.liu@gmail.com>
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi, Feng
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for driving this FLIP, this proposal is very
>> > useful
>> > > > for
>> > > > > > > > catalog
>> > > > > > > > > > > management.
>> > > > > > > > > > > I have some small questions:
>> > > > > > > > > > >
>> > > > > > > > > > > 1. Regarding the
>> CatalogStoreFactory#createCatalogStore
>> > > > method,
>> > > > > > do we
>> > > > > > > > > > need
>> > > > > > > > > > > to provide a default implementation?
>> > > > > > > > > > > 2. If we get Catalog from CatalogStore, after
>> > initializing
>> > > > it,
>> > > > > > > > whether
>> > > > > > > > > we
>> > > > > > > > > > > put it to Map<String, Catalog> catalogs again?
>> > > > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
>> > > > > > > > > > > `sql.catalog.store.file.path`, how about renaming
>> them to
>> > > > > > > > > > > `catalog.store.type` and `catalog.store.path`?
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Ron
>> > > > > > > > > > >
>> > > > > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一
>> 21:19写道：
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi yuxia
>> > > > > > > > > > > >
>> > > > > > > > > > > > > But from the code in Proposed Changes, once we
>> > > register
>> > > > > the
>> > > > > > > > > Catalog,
>> > > > > > > > > > > we
>> > > > > > > > > > > > initialize it and open it. right?
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
>> > > > original
>> > > > > > > > CREATE
>> > > > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
>> > > > > > > > registerCatalog
>> > > > > > > > > so
>> > > > > > > > > > > > that parameter validation can be performed.
>> > > > > > > > > > > >
>> > > > > > > > > > > > In the current design, lazy initialization is mainly
>> > > > > reflected
>> > > > > > in
>> > > > > > > > > > > > getCatalog. If CatalogStore has already saved some
>> > > catalog
>> > > > > > > > > > > configurations,
>> > > > > > > > > > > > only initialization is required in getCatalog.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Best,
>> > > > > > > > > > > > Feng
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
>> > > > > > luoyuxia@alumni.sjtu.edu.cn
>> > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi, Feng.
>> > > > > > > > > > > > > I'm trying to understanding the meaning of *lazy
>> > > > > > initialization*.
>> > > > > > > > > If
>> > > > > > > > > > > i'm
>> > > > > > > > > > > > > wrong, please correct me.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > IIUC, lazy initialization means only you need to
>> > access
>> > > > the
>> > > > > > > > > catalog,
>> > > > > > > > > > > then
>> > > > > > > > > > > > > you initialize it. But from the code in Proposed
>> > > Changes,
>> > > > > > once we
>> > > > > > > > > > > > register
>> > > > > > > > > > > > > the Catalog,
>> > > > > > > > > > > > > we initialize it and open it. right?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > Yuxia
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > ----- 原始邮件 -----
>> > > > > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
>> > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
>> > > > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
>> > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
>> > > > Catalog
>> > > > > > > > > > > configuration
>> > > > > > > > > > > > > and asynchronous registration
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for your effort! +1 for the proposal.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > One of the major changes is that current design
>> will
>> > > > > provide
>> > > > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead
>> > of a
>> > > > > > cache,
>> > > > > > > > > which
>> > > > > > > > > > > > means
>> > > > > > > > > > > > > once it has been initialized, any changes done by
>> > other
>> > > > > > sessions
>> > > > > > > > > will
>> > > > > > > > > > > not
>> > > > > > > > > > > > > affect it. Point 6 described follow-up options for
>> > > > further
>> > > > > > > > > > improvement.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > Jing
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
>> > > > > > jinfeng1995@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi all, I would like to update you on the latest
>> > > > progress
>> > > > > > of
>> > > > > > > > the
>> > > > > > > > > > > FLIP.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge,
>> Shammon
>> > FY,
>> > > > > > ShengKai
>> > > > > > > > > Fang
>> > > > > > > > > > > > and I
>> > > > > > > > > > > > > > had an offline discussion regarding the overall
>> > > > solution
>> > > > > > for
>> > > > > > > > > Flink
>> > > > > > > > > > > > > > CatalogStore. We have reached a consensus and I
>> > have
>> > > > > > updated
>> > > > > > > > the
>> > > > > > > > > > > final
>> > > > > > > > > > > > > > solution in FLIP.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Next, let me briefly describe the entire design:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 1.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
>> > > > > > configuration
>> > > > > > > > > > similar
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > TableDescriptor.
>> > > > > > > > > > > > > > 2.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > The two key functions of CatalogStore - void
>> > > > > > > > > storeCatalog(String
>> > > > > > > > > > > > > > catalogName, CatalogDescriptor) and
>> > > > CatalogDescriptor
>> > > > > > > > > > > > > getCatalog(String)
>> > > > > > > > > > > > > > will both use CatalogDescriptor instead of
>> > Catalog
>> > > > > > instance.
>> > > > > > > > > > This
>> > > > > > > > > > > > way,
>> > > > > > > > > > > > > > CatalogStore will only be responsible for
>> saving
>> > > and
>> > > > > > > > > retrieving
>> > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > configurations without having to initialize
>> > > > catalogs.
>> > > > > > > > > > > > > > 3.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > The default registerCatalog(String
>> catalogName,
>> > > > > Catalog
>> > > > > > > > > catalog)
>> > > > > > > > > > > > > > function in CatalogManager will be marked as
>> > > > > deprecated.
>> > > > > > > > > > > > > > 4.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > A new function registerCatalog(String
>> > catalogName,
>> > > > > > > > > > > CatalogDescriptor
>> > > > > > > > > > > > > > catalog) will be added to serve as the
>> default
>> > > > > > registration
>> > > > > > > > > > > function
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > catalogs in CatalogManager.
>> > > > > > > > > > > > > > 5.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager
>> > will
>> > > > > > remain
>> > > > > > > > > > > unchanged
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > save initialized catalogs.This means that
>> > deletion
>> > > > > > > > operations
>> > > > > > > > > > from
>> > > > > > > > > > > > one
>> > > > > > > > > > > > > > session won't synchronize with other
>> sessions.
>> > > > > > > > > > > > > > 6.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > To support multi-session synchronization
>> > scenarios
>> > > > for
>> > > > > > > > > deletions
>> > > > > > > > > > > > later
>> > > > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
>> > > > > > > > > configurable.There
>> > > > > > > > > > > may
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > three possible situations:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > a.Default caching of all initialized catalogs
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > b.Introduction of LRU cache logic which can
>> > > > > > automatically
>> > > > > > > > > clear
>> > > > > > > > > > > > > > long-unused catalogs.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > c.No caching of any instances; each call to
>> > > > getCatalog
>> > > > > > > > > creates a
>> > > > > > > > > > > new
>> > > > > > > > > > > > > > instance.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This is the document for discussion:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This is the final proposal document:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thank you very much for your attention and
>> > > suggestions
>> > > > on
>> > > > > > this
>> > > > > > > > > > > FLIP. A
>> > > > > > > > > > > > > > special thanks to Hang Ruan for his suggestions
>> on
>> > > the
>> > > > > > entire
>> > > > > > > > > > design
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > organizing offline discussions.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > If you have any further suggestions or feedback
>> > about
>> > > > > this
>> > > > > > FLIP
>> > > > > > > > > > > please
>> > > > > > > > > > > > > feel
>> > > > > > > > > > > > > > free to share.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
>> > > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Thanks for improving the FLIP. It looks good
>> to
>> > me.
>> > > > We
>> > > > > > could
>> > > > > > > > > > still
>> > > > > > > > > > > > > > > reconsider in the future how to provide more
>> > common
>> > > > > > built-in
>> > > > > > > > > > cache
>> > > > > > > > > > > > > > > functionality in CatalogManager so that not
>> every
>> > > > > > > > CatalogSotre
>> > > > > > > > > > > > > > > implementation has to take care of it.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
>> > > > > > > > jinfeng1995@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Hi Jing,
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Thanks for your reply.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > There might be more such issues. I would
>> > > suggest
>> > > > > you
>> > > > > > > > > > > completely
>> > > > > > > > > > > > > walk
>> > > > > > > > > > > > > > > > through the FLIP again and fix those issues
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > I am very sorry for my carelessness and at
>> the
>> > > same
>> > > > > > time, I
>> > > > > > > > > > > greatly
>> > > > > > > > > > > > > > > > appreciate your careful review. I have
>> > thoroughly
>> > > > > > checked
>> > > > > > > > the
>> > > > > > > > > > > > entire
>> > > > > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > > and made corrections to these issues.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > If I am not mistaken, with the current
>> FLIP
>> > > > > design,
>> > > > > > > > > > > > > CatalogManager
>> > > > > > > > > > > > > > > > could work without Optional CatalogStore
>> being
>> > > > > > configured.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Yes, in the original design, CatalogStore
>> was
>> > not
>> > > > > > necessary
>> > > > > > > > > > > because
>> > > > > > > > > > > > > > > > CatalogManager used Map<String, Catalog>
>> > catalogs
>> > > > to
>> > > > > > store
>> > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > instances.
>> > > > > > > > > > > > > > > > However, this caused inconsistency issues.
>> > > > > Therefore, I
>> > > > > > > > > > modified
>> > > > > > > > > > > > this
>> > > > > > > > > > > > > > > part
>> > > > > > > > > > > > > > > > of the design and removed Map<String,
>> Catalog>
>> > > > > catalogs
>> > > > > > > > from
>> > > > > > > > > > > > > > > > CatalogManager.
>> > > > > > > > > > > > > > > > At the same time, InMemoryCatalog will
>> serve
>> > as
>> > > > the
>> > > > > > > > default
>> > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > to save catalogs in memory and replace the
>> > > > > > functionality of
>> > > > > > > > > > > > > > > > Map<String,Catalog>catalogs.
>> > > > > > > > > > > > > > > > The previous plan that kept
>> > > > > > Map<String,Catalog>catalogs has
>> > > > > > > > > > been
>> > > > > > > > > > > > > moved
>> > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > Rejected Alternatives.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
>> > > > > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > There are still many places contain
>> > > inconsistent
>> > > > > > content,
>> > > > > > > > > > e.g.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 1. "Asynchronous registration" is still
>> used.
>> > > > > > > > > > > > > > > > > 2. The java comment of the method
>> > > > > > registerCatalog(String
>> > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does
>> not
>> > > tell
>> > > > > > what the
>> > > > > > > > > > > method
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > doing.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > There might be more such issues. I would
>> > > suggest
>> > > > > you
>> > > > > > > > > > completely
>> > > > > > > > > > > > > walk
>> > > > > > > > > > > > > > > > > through the FLIP again and fix those
>> issues.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you
>> mean
>> > > that
>> > > > > you
>> > > > > > will
>> > > > > > > > > > build
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > cache
>> > > > > > > > > > > > > > > > > functionality in the CatalogStore? This
>> is a
>> > > very
>> > > > > > > > different
>> > > > > > > > > > > > design
>> > > > > > > > > > > > > > > > concept
>> > > > > > > > > > > > > > > > > from what the current FLIP described. If
>> I am
>> > > not
>> > > > > > > > mistaken,
>> > > > > > > > > > > with
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > current FLIP design, CatalogManager could
>> > work
>> > > > > > without
>> > > > > > > > > > Optional
>> > > > > > > > > > > > > > > > > CatalogStore being configured. That is the
>> > > reason
>> > > > > > why I
>> > > > > > > > > > > mentioned
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > last email that the example code wrt the
>> > > Optional
>> > > > > > > > > > CatalogStore
>> > > > > > > > > > > is
>> > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > correct.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
>> > > > > > > > > > > jinfeng1995@gmail.com>
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Hi Jing
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > There are still some NIT issues in the
>> > FLIP
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Thank you very much for the careful
>> > review. I
>> > > > > have
>> > > > > > > > > already
>> > > > > > > > > > > made
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > relevant changes.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in
>> the
>> > > > > > > > multi-instance
>> > > > > > > > > > > > > > scenarios, I
>> > > > > > > > > > > > > > > > am
>> > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > sure if this is the intended behaviour
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Currently, there are conflicts in
>> multiple
>> > > > > > scenarios
>> > > > > > > > with
>> > > > > > > > > > the
>> > > > > > > > > > > > > > current
>> > > > > > > > > > > > > > > > > > design. I am thinking whether we should
>> > > remove
>> > > > > > > > > 'Map<String,
>> > > > > > > > > > > > > > Catalog>'
>> > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > make Cache the default behavior of
>> > > > > > > > InMemoryCatalogStore.
>> > > > > > > > > > This
>> > > > > > > > > > > > > way,
>> > > > > > > > > > > > > > > > users
>> > > > > > > > > > > > > > > > > > can implement their own CatalogStore to
>> > > achieve
>> > > > > > > > > > > multi-instance
>> > > > > > > > > > > > > > > > > > non-conflicting scenarios. What do you
>> > think?
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
>> > > > > > > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There
>> are
>> > > > still
>> > > > > > some
>> > > > > > > > > NIT
>> > > > > > > > > > > > issues
>> > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > > > > > like:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore
>> > has
>> > > > been
>> > > > > > used
>> > > > > > > > as
>> > > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > > instead of Optional in the code
>> example.
>> > It
>> > > > > > should be
>> > > > > > > > > > fine
>> > > > > > > > > > > to
>> > > > > > > > > > > > > use
>> > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > pseudo code for now and update it
>> after
>> > you
>> > > > > > submit
>> > > > > > > > the
>> > > > > > > > > > PR.
>> > > > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used
>> > somewhere
>> > > in
>> > > > > the
>> > > > > > > > > > rejected
>> > > > > > > > > > > > > > section
>> > > > > > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > > > > > should be persistContext(...) to keep
>> it
>> > > > > > consistent.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
>> > > > > > multi-instance
>> > > > > > > > > > > > > scenarios,
>> > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > am
>> > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > sure if this is the intended
>> behaviour.
>> > If
>> > > > > > > > Map<String,
>> > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > catalogs
>> > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > used as a cache, it should be invalid,
>> > once
>> > > > the
>> > > > > > > > related
>> > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > has
>> > > > > > > > > > > > > > > > > been
>> > > > > > > > > > > > > > > > > > > removed from the CatalogStore by
>> another
>> > > > > > instance.
>> > > > > > > > Did
>> > > > > > > > > I
>> > > > > > > > > > > miss
>> > > > > > > > > > > > > > > > > something?
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng
>> Jin
>> > <
>> > > > > > > > > > > > > jinfeng1995@gmail.com>
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Hi Jing,Shammon
>> > > > > > > > > > > > > > > > > > > > Thanks for your reply.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > @Jing
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > How about persistCatalog()?
>> > > > > > > > > > > > > > > > > > > > I think this is a good function
>> name, I
>> > > > have
>> > > > > > > > updated
>> > > > > > > > > it
>> > > > > > > > > > > in
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > documentation.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Some common cache features
>> should
>> > be
>> > > > > > > > implemented
>> > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
>> > > > alternative
>> > > > > 1
>> > > > > > > > turns
>> > > > > > > > > > out
>> > > > > > > > > > > to
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > appropriate later, I will improve
>> this
>> > > part
>> > > > > of
>> > > > > > the
>> > > > > > > > > > > content.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > As the above discussion moves
>> > forward,
>> > > > the
>> > > > > > > > option 2
>> > > > > > > > > > > > > solution
>> > > > > > > > > > > > > > > > looks
>> > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > like a replacement of option 1
>> > > > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
>> > > offline,
>> > > > > we
>> > > > > > > > think
>> > > > > > > > > > that
>> > > > > > > > > > > > > > > solution
>> > > > > > > > > > > > > > > > 2
>> > > > > > > > > > > > > > > > > > > might
>> > > > > > > > > > > > > > > > > > > > be more suitable and also avoid any
>> > > > > > inconsistency
>> > > > > > > > > > issues.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > There are some inconsistent
>> > > descriptions
>> > > > in
>> > > > > > the
>> > > > > > > > > > > content.
>> > > > > > > > > > > > > > Would
>> > > > > > > > > > > > > > > > you
>> > > > > > > > > > > > > > > > > > > like
>> > > > > > > > > > > > > > > > > > > > to clean them up?
>> > > > > > > > > > > > > > > > > > > > I will do my best to improve the
>> > document
>> > > > and
>> > > > > > > > > > appreciate
>> > > > > > > > > > > > your
>> > > > > > > > > > > > > > > > > > > suggestions.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > @Shammon
>> > > > > > > > > > > > > > > > > > > > > can you put the unselected option
>> in
>> > > > > > `Rejected
>> > > > > > > > > > > > > Alternatives`
>> > > > > > > > > > > > > > > > > > > > Sure, I have moved it to the
>> `Rejected
>> > > > > > > > Alternatives`.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Best
>> > > > > > > > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM
>> Shammon
>> > > FY
>> > > > <
>> > > > > > > > > > > > > zjureel@gmail.com>
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Hi Feng
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Thanks for your update.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > I found there are two options in
>> > > > `Proposed
>> > > > > > > > > Changes`,
>> > > > > > > > > > > can
>> > > > > > > > > > > > > you
>> > > > > > > > > > > > > > > put
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > unselected option in `Rejected
>> > > > > > Alternatives`? I
>> > > > > > > > > think
>> > > > > > > > > > > > this
>> > > > > > > > > > > > > > may
>> > > > > > > > > > > > > > > > help
>> > > > > > > > > > > > > > > > > > us
>> > > > > > > > > > > > > > > > > > > > > better understand your proposal
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > Shammon FY
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM
>> Jing
>> > Ge
>> > > > > > > > > > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I
>> am
>> > > > still
>> > > > > > > > confused
>> > > > > > > > > > > after
>> > > > > > > > > > > > > > > > > completely
>> > > > > > > > > > > > > > > > > > > > > reading
>> > > > > > > > > > > > > > > > > > > > > > the thread with following
>> > questions:
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
>> > > registerCatalog()
>> > > > > and
>> > > > > > > > > > > > addCatalog()
>> > > > > > > > > > > > > > have
>> > > > > > > > > > > > > > > > no
>> > > > > > > > > > > > > > > > > > big
>> > > > > > > > > > > > > > > > > > > > > > difference based on their names.
>> > One
>> > > of
>> > > > > > them is
>> > > > > > > > > > > > > responsible
>> > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > data
>> > > > > > > > > > > > > > > > > > > > > > persistence. How about
>> > > > persistCatalog()?
>> > > > > > > > > > > > > > > > > > > > > > 2. As you mentioned that
>> > Map<String,
>> > > > > > Catalog>
>> > > > > > > > > > > catalogs
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > > used
>> > > > > > > > > > > > > > > > > as a
>> > > > > > > > > > > > > > > > > > > > cache
>> > > > > > > > > > > > > > > > > > > > > > and catalogStore is used for
>> data
>> > > > > > persistence.
>> > > > > > > > I
>> > > > > > > > > > > would
>> > > > > > > > > > > > > > > suggest
>> > > > > > > > > > > > > > > > > > > > describing
>> > > > > > > > > > > > > > > > > > > > > > their purpose conceptually and
>> > > clearly
>> > > > in
>> > > > > > the
>> > > > > > > > > FLIP.
>> > > > > > > > > > > > Some
>> > > > > > > > > > > > > > > common
>> > > > > > > > > > > > > > > > > > cache
>> > > > > > > > > > > > > > > > > > > > > > features should be implemented,
>> > i.e.
>> > > > data
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > > cache
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > store
>> > > > > > > > > > > > > > > > > > > > > should
>> > > > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog
>> > instance
>> > > > > > should be
>> > > > > > > > > > found
>> > > > > > > > > > > in
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > store
>> > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > the cache(either it has been
>> > > > initialized
>> > > > > > or it
>> > > > > > > > > will
>> > > > > > > > > > > be
>> > > > > > > > > > > > > lazy
>> > > > > > > > > > > > > > > > > > > > initialized)
>> > > > > > > > > > > > > > > > > > > > > > for the same catalog name. The
>> > > > > consistency
>> > > > > > will
>> > > > > > > > > be
>> > > > > > > > > > > > taken
>> > > > > > > > > > > > > > care
>> > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > while
>> > > > > > > > > > > > > > > > > > > > > > updating the catalog.
>> > > > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
>> > > > forward,
>> > > > > > the
>> > > > > > > > > > option
>> > > > > > > > > > > 2
>> > > > > > > > > > > > > > > solution
>> > > > > > > > > > > > > > > > > > looks
>> > > > > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
>> > > > because,
>> > > > > > afaiu,
>> > > > > > > > > > > issues
>> > > > > > > > > > > > > > > > mentioned
>> > > > > > > > > > > > > > > > > > > > > > previously with option 1 are not
>> > > solved
>> > > > > > yet. Do
>> > > > > > > > > you
>> > > > > > > > > > > > still
>> > > > > > > > > > > > > > > want
>> > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > propose
>> > > > > > > > > > > > > > > > > > > > > > both options and ask for
>> > suggestions
>> > > > for
>> > > > > > both
>> > > > > > > > of
>> > > > > > > > > > > them?
>> > > > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP,
>> > there
>> > > > are
>> > > > > > some
>> > > > > > > > > > > > > inconsistent
>> > > > > > > > > > > > > > > > > > > descriptions
>> > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > the content. Would you like to
>> > clean
>> > > > > them
>> > > > > > up?
>> > > > > > > > > > > Thanks!
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM
>> Feng
>> > > > Jin <
>> > > > > > > > > > > > > > > jinfeng1995@gmail.com
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > hi Shammon
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > Thank you for your response,
>> and
>> > I
>> > > > > > completely
>> > > > > > > > > > agree
>> > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > your
>> > > > > > > > > > > > > > > > > > point
>> > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > view.
>> > > > > > > > > > > > > > > > > > > > > > > Initially, I may have over
>> > > > complicated
>> > > > > > the
>> > > > > > > > > whole
>> > > > > > > > > > > > issue.
>> > > > > > > > > > > > > > > First
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > foremost,
>> > > > > > > > > > > > > > > > > > > > > > > we need to consider the
>> > persistence
>> > > > of
>> > > > > > the
>> > > > > > > > > > > Catalog's
>> > > > > > > > > > > > > > > > > > Configuration.
>> > > > > > > > > > > > > > > > > > > > > > > If we only need to provide
>> > > > persistence
>> > > > > > for
>> > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > Configuration,
>> > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > add a toConfiguration method
>> to
>> > the
>> > > > > > Catalog
>> > > > > > > > > > > > interface.
>> > > > > > > > > > > > > > > > > > > > > > > This method can convert a
>> Catalog
>> > > > > > instance
>> > > > > > > > to a
>> > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > String>
>> > > > > > > > > > > > > > > > > > > > > > > properties, and the default
>> > > > > > implementation
>> > > > > > > > will
>> > > > > > > > > > > throw
>> > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > exception.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > public interface Catalog {
>> > > > > > > > > > > > > > > > > > > > > > > /**
>> > > > > > > > > > > > > > > > > > > > > > > * Returns a map containing
>> the
>> > > > > > properties
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > object.
>> > > > > > > > > > > > > > > > > > > > > > > *
>> > > > > > > > > > > > > > > > > > > > > > > * @return a map containing
>> the
>> > > > > > properties
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > object
>> > > > > > > > > > > > > > > > > > > > > > > * @throws
>> > > > > > UnsupportedOperationException if
>> > > > > > > > > the
>> > > > > > > > > > > > > > > > implementing
>> > > > > > > > > > > > > > > > > > > class
>> > > > > > > > > > > > > > > > > > > > > does
>> > > > > > > > > > > > > > > > > > > > > > > not override
>> > > > > > > > > > > > > > > > > > > > > > > * the default
>> implementation
>> > of
>> > > > this
>> > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > > > */
>> > > > > > > > > > > > > > > > > > > > > > > default Map<String, String>
>> > > > > > toProperties()
>> > > > > > > > {
>> > > > > > > > > > > > > > > > > > > > > > > throw new
>> > > > > > > > > > UnsupportedOperationException("Please
>> > > > > > > > > > > > > > > implement
>> > > > > > > > > > > > > > > > > > > > > > toProperties
>> > > > > > > > > > > > > > > > > > > > > > > for this catalog");
>> > > > > > > > > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > The specific process is as
>> > follows:
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > 1. If the user has
>> configured a
>> > > > > > > > CatalogStore,
>> > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > toConfiguration()
>> > > > > > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > > > will be called when
>> registering a
>> > > > > Catalog
>> > > > > > > > > > instance
>> > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
>> > catalogName,
>> > > > > > Catalog
>> > > > > > > > > > > catalog).
>> > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > > > > > > > > will be converted to a
>> > Map<String,
>> > > > > > String>
>> > > > > > > > > > > properties
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > saved
>> > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
>> > > > instances
>> > > > > > do
>> > > > > > > > not
>> > > > > > > > > > > > > implement
>> > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > method,
>> > > > > > > > > > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > > > > > > exception will be thrown.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
>> > > > > > CatalogStore,
>> > > > > > > > > the
>> > > > > > > > > > > > > > > > > > toConfiguration()
>> > > > > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
>> > > > > consistency
>> > > > > > with
>> > > > > > > > > the
>> > > > > > > > > > > > > > original
>> > > > > > > > > > > > > > > > > > process.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
>> > > > Catalog>
>> > > > > > > > > catalogs
>> > > > > > > > > > > and
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > > > > at
>> > > > > > > > > > > > > > > > > > > > > > > the same time can also avoid
>> > > > conflicts
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > For lazy initialization:
>> > > > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
>> > > itself
>> > > > > and
>> > > > > > > > > provide
>> > > > > > > > > > a
>> > > > > > > > > > > > > > > dedicated
>> > > > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > > implementation for lazy
>> loading,
>> > > such
>> > > > > as
>> > > > > > > > > > > > > > > ConfigurationCatalog
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > public class
>> ConfigurationCatalog
>> > > > > > implements
>> > > > > > > > > > > Catalog
>> > > > > > > > > > > > {
>> > > > > > > > > > > > > > > > > > > > > > >
>> ConfigurationCatalog(Map<String,
>> > > > > String>
>> > > > > > > > > > > properties)
>> > > > > > > > > > > > {
>> > > > > > > > > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > I have added this design to
>> the
>> > > FLIP.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at
>> 10:03 AM
>> > > > Shammon
>> > > > > > FY <
>> > > > > > > > > > > > > > > > zjureel@gmail.com>
>> > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > I have no questions about
>> the
>> > > > > > functionality
>> > > > > > > > > of
>> > > > > > > > > > > > > > > > > `CatalogStore`,
>> > > > > > > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > different operations of
>> > multiple
>> > > > > > > > > > > `registerCatalog`
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > `storeCatalog`
>> > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
>> > > > implementation
>> > > > > in
>> > > > > > > > Trino
>> > > > > > > > > > is
>> > > > > > > > > > > > also
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > same,
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino
>> only
>> > > has
>> > > > > one
>> > > > > > > > > > > > > `createCatalog`,
>> > > > > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > save
>> > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > catalog to memory and
>> > > > `CatalogStore`.
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
>> > > > another
>> > > > > > > > > > > > > `registerCatalog`
>> > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > `addCatalog`
>> > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > save a catalog in
>> `Map<String,
>> > > > > Catalog>
>> > > > > > > > > > catalogs`
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > `CatalogStore
>> > > > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you
>> > mentioned
>> > > > > > before,
>> > > > > > > > the
>> > > > > > > > > > > > > > `Map<String,
>> > > > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
>> > > > > > `CatalogManager`.
>> > > > > > > > How
>> > > > > > > > > > > about
>> > > > > > > > > > > > > > > saving
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog>
>> catalogs`
>> > > and
>> > > > > > > > > > `CatalogStore
>> > > > > > > > > > > > > > > > > catalogStore`
>> > > > > > > > > > > > > > > > > > > > > > together
>> > > > > > > > > > > > > > > > > > > > > > > > when it is registered or
>> added
>> > in
>> > > > > > > > > > > `CatalogManager`?
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
>> 5:22 PM
>> > > Feng
>> > > > > > Jin <
>> > > > > > > > > > > > > > > > > jinfeng1995@gmail.com
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply.
>> I
>> > am
>> > > > very
>> > > > > > sorry
>> > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > misunderstanding
>> > > > > > > > > > > > > > > > > > > > > > > caused
>> > > > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
>> > > original
>> > > > > > > > > discussion.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
>> > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
>> > > > pre-discussion
>> > > > > > [1]
>> > > > > > > > for
>> > > > > > > > > > > this
>> > > > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed
>> such a
>> > > > > > discussion
>> > > > > > > > > > before.
>> > > > > > > > > > > > > > > However,
>> > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > designing
>> > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > whole solution, I found
>> that
>> > > the
>> > > > > > logic of
>> > > > > > > > > > > > > > > CatalogManager
>> > > > > > > > > > > > > > > > > > itself
>> > > > > > > > > > > > > > > > > > > > > > doesn't
>> > > > > > > > > > > > > > > > > > > > > > > > > need to change much. *We
>> > cannot
>> > > > > only
>> > > > > > > > > persist
>> > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > instances
>> > > > > > > > > > > > > > > > > > > > > > > > themselves*,
>> > > > > > > > > > > > > > > > > > > > > > > > > so exposing only
>> > > > > > registerCatalog(String
>> > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > > > > > > > might
>> > > > > > > > > > > > > > > > > > > > > > > > > not be enough to save
>> > Catalogs,
>> > > > > > because
>> > > > > > > > in
>> > > > > > > > > > the
>> > > > > > > > > > > > end
>> > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > still
>> > > > > > > > > > > > > > > > > > > need
>> > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > save
>> > > > > > > > > > > > > > > > > > > > > > > > > the configurations
>> > > corresponding
>> > > > to
>> > > > > > the
>> > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > instances.
>> > > > > > > > > > > > > > > > > > > > > > Therefore,
>> > > > > > > > > > > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
>> > > > > CatalogStore
>> > > > > > > > > > interface
>> > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > configuration
>> > > > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding
>> this
>> > > > part, I
>> > > > > > also
>> > > > > > > > > took
>> > > > > > > > > > > > > > > inspiration
>> > > > > > > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > > > > > > Trino's
>> > > > > > > > > > > > > > > > > > > > > > > > > implementation[1].
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > @yuxia
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
>> > handling
>> > > > > > catalog
>> > > > > > > > with
>> > > > > > > > > > the
>> > > > > > > > > > > > > same
>> > > > > > > > > > > > > > > name
>> > > > > > > > > > > > > > > > > > > looks a
>> > > > > > > > > > > > > > > > > > > > > > > little
>> > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > complex to me.
>> > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the
>> > suggestion. I
>> > > > > will
>> > > > > > > > > provide
>> > > > > > > > > > a
>> > > > > > > > > > > > > > detailed
>> > > > > > > > > > > > > > > > > > > > description
>> > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > code examples for this
>> part,
>> > > and
>> > > > > add
>> > > > > > it
>> > > > > > > > to
>> > > > > > > > > > the
>> > > > > > > > > > > > > FLIP.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
>> > > > > > `addCatalog`
>> > > > > > > > > still
>> > > > > > > > > > > > looks
>> > > > > > > > > > > > > > > > confused
>> > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > me.
>> > > > > > > > > > > > > > > > > > > > > > > IIUC,
>> > > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
>> > > > catalog
>> > > > > to
>> > > > > > > > > > > > CatalogStore,
>> > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > > about
>> > > > > > > > > > > > > > > > > > > > > > renaming
>> > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
>> very
>> > > > > personal
>> > > > > > > > > > opinion,
>> > > > > > > > > > > > you
>> > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > decide
>> > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > take
>> > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
>> > > > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
>> > > intuitive
>> > > > > to
>> > > > > > me.
>> > > > > > > > I
>> > > > > > > > > > > don't
>> > > > > > > > > > > > > see
>> > > > > > > > > > > > > > > any
>> > > > > > > > > > > > > > > > > > > problem
>> > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > it.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
>> > > statement,
>> > > > > > which
>> > > > > > > > > > method
>> > > > > > > > > > > > will
>> > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > called?
>> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
>> > > > `addCatalog`?
>> > > > > > I'm
>> > > > > > > > > > > wondering
>> > > > > > > > > > > > > > > whether
>> > > > > > > > > > > > > > > > > user
>> > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > add a
>> > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
>> > > > stement.
>> > > > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my
>> > original
>> > > > > > design
>> > > > > > > > was
>> > > > > > > > > to
>> > > > > > > > > > > add
>> > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > directly
>> > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this
>> would
>> > > > > disrupt
>> > > > > > the
>> > > > > > > > > > > existing
>> > > > > > > > > > > > > > > logic.
>> > > > > > > > > > > > > > > > > > > > > Therefore, I
>> > > > > > > > > > > > > > > > > > > > > > > > think
>> > > > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
>> > > > > > configuration to
>> > > > > > > > > the
>> > > > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > initialize
>> > > > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at
>> the
>> > > same
>> > > > > time
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really
>> neccessary
>> > to
>> > > > > > provide a
>> > > > > > > > > > > default
>> > > > > > > > > > > > > > > > > > implmentation
>> > > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > > > interface
>> > > `CatalogStoreFactory`?
>> > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
>> > > > otherwise
>> > > > > we
>> > > > > > > > would
>> > > > > > > > > > > need
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > introduce
>> > > > > > > > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > > > > > > > additional
>> > > > > > > > > > > > > > > > > > > > > > > > > Map to store the
>> > configuration
>> > > > for
>> > > > > > lazy
>> > > > > > > > > > > loading.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
>> > > > > registration
>> > > > > > for
>> > > > > > > > > > > catalog.
>> > > > > > > > > > > > > > > > > > > > > > > > > I don't think
>> > > > > registerCatalog(String
>> > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
>> > > > interface
>> > > > > > > > because
>> > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > already
>> > > > > > > > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > > > > > > > instance.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
>> > > > > > initialization
>> > > > > > > > > for
>> > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > than
>> > > > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
>> > > > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
>> > > > > > inaccurate. It
>> > > > > > > > > > > should
>> > > > > > > > > > > > be
>> > > > > > > > > > > > > > > lazy
>> > > > > > > > > > > > > > > > > > > > > registration
>> > > > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
>> > > > > > registration. I
>> > > > > > > > > have
>> > > > > > > > > > > > > already
>> > > > > > > > > > > > > > > > > updated
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > title
>> > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > the FLIP.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > [1].
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
>> > 4:27 PM
>> > > > > > Shammon
>> > > > > > > > FY <
>> > > > > > > > > > > > > > > > > zjureel@gmail.com
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
>> > > > > > `registerCatalog`
>> > > > > > > > > > > method
>> > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > `CatalogManager`,
>> > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
>> > > method
>> > > > > > named
>> > > > > > > > > > > > `addCatalog`
>> > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > `storeCatalog`
>> > > > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > > added.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned,
>> the
>> > > > memory
>> > > > > > > > catalog
>> > > > > > > > > > is a
>> > > > > > > > > > > > > > > `cache`,
>> > > > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > > > > > think
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > concept
>> > > > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
>> > > > exposed
>> > > > > to
>> > > > > > > > > users.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
>> > > > pre-discussion
>> > > > > > [1]
>> > > > > > > > for
>> > > > > > > > > > > this
>> > > > > > > > > > > > > > FLIP.
>> > > > > > > > > > > > > > > > > Please
>> > > > > > > > > > > > > > > > > > > > > correct
>> > > > > > > > > > > > > > > > > > > > > > > me
>> > > > > > > > > > > > > > > > > > > > > > > > if
>> > > > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
>> > > conclusion
>> > > > > of
>> > > > > > that
>> > > > > > > > > > > > > discussion
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > use
>> > > > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
>> > > > interface
>> > > > > > and
>> > > > > > > > > > > implement
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > different
>> > > > > > > > > > > > > > > > > > > > > > > stores
>> > > > > > > > > > > > > > > > > > > > > > > > > such
>> > > > > > > > > > > > > > > > > > > > > > > > > > as memory, file and
>> > external
>> > > > > > system.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
>> > > between
>> > > > > the
>> > > > > > > > > current
>> > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > design
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > > > > > > > > conclusion.
>> > > > > > > > > > > > > > > > > > > > > > > > > > What about the proposal
>> of
>> > > the
>> > > > > > > > discussion
>> > > > > > > > > > in
>> > > > > > > > > > > > > thread
>> > > > > > > > > > > > > > > > [1] ?
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > [1]
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > >
>> > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
>> > > 3:41 PM
>> > > > > > yuxia <
>> > > > > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
>> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for
>> driving
>> > > this
>> > > > > > FLIP.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
>> > > handling
>> > > > > > catalog
>> > > > > > > > > with
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > same
>> > > > > > > > > > > > > > > > name
>> > > > > > > > > > > > > > > > > > > > looks a
>> > > > > > > > > > > > > > > > > > > > > > > > little
>> > > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
>> > > it'll
>> > > > be
>> > > > > > > > better
>> > > > > > > > > to
>> > > > > > > > > > > > > explain
>> > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > java
>> > > > > > > > > > > > > > > > > > > > > > > doc
>> > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > these methods and
>> give a
>> > > > brief
>> > > > > > > > example
>> > > > > > > > > in
>> > > > > > > > > > > > this
>> > > > > > > > > > > > > > > FLIP.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method
>> name
>> > > > > > `addCatalog`
>> > > > > > > > > > still
>> > > > > > > > > > > > > looks
>> > > > > > > > > > > > > > > > > confused
>> > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > me.
>> > > > > > > > > > > > > > > > > > > > > > > > IIUC,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > this method is for
>> > storing
>> > > > > > catalog to
>> > > > > > > > > > > > > > CatalogStore,
>> > > > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > > > > about
>> > > > > > > > > > > > > > > > > > > > > > > > renaming
>> > > > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`?
>> It's
>> > > very
>> > > > > > personal
>> > > > > > > > > > > > opinion,
>> > > > > > > > > > > > > > you
>> > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > decide
>> > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > take
>> > > > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
>> > > > > statement,
>> > > > > > > > which
>> > > > > > > > > > > method
>> > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > called?
>> > > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
>> > > > > > `addCatalog`?
>> > > > > > > > I'm
>> > > > > > > > > > > > > wondering
>> > > > > > > > > > > > > > > > > whether
>> > > > > > > > > > > > > > > > > > > user
>> > > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > > add a
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with
>> SQL
>> > > > > > stement.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really
>> > neccessary
>> > > to
>> > > > > > > > provide a
>> > > > > > > > > > > > default
>> > > > > > > > > > > > > > > > > > > implmentation
>> > > > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > > > > > interface
>> > > > > `CatalogStoreFactory`?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
>> > > > > > registration
>> > > > > > > > for
>> > > > > > > > > > > > catalog.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > When creating a
>> catalog
>> > > > with
>> > > > > > CREATE
>> > > > > > > > > > > > CATALOG,
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
>> > used
>> > > > by
>> > > > > > > > default.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
>> > > registration
>> > > > is
>> > > > > > the
>> > > > > > > > > > default
>> > > > > > > > > > > > > > > behavior,
>> > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > there
>> > > > > > > > > > > > > > > > > > > > > > any
>> > > > > > > > > > > > > > > > > > > > > > > > way
>> > > > > > > > > > > > > > > > > > > > > > > > > > > that user can switch
>> to
>> > > > > > synchronous
>> > > > > > > > > > > > > registration
>> > > > > > > > > > > > > > > just
>> > > > > > > > > > > > > > > > > > like
>> > > > > > > > > > > > > > > > > > > > > > before?
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Will both method
>> > > `addCatalog`
>> > > > > and
>> > > > > > > > > > > > > > `registerCatalog`
>> > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registration?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
>> > > > > > registration,
>> > > > > > > > it
>> > > > > > > > > > may
>> > > > > > > > > > > > well
>> > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > CREATE
>> > > > > > > > > > > > > > > > > > > > > > CATALOG
>> > > > > > > > > > > > > > > > > > > > > > > > > > > executes successfully,
>> > but
>> > > > then
>> > > > > > the
>> > > > > > > > > > > following
>> > > > > > > > > > > > > > > CREATE
>> > > > > > > > > > > > > > > > > > TABLE
>> > > > > > > > > > > > > > > > > > > > > > > statement
>> > > > > > > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog
>> fail
>> > > to
>> > > > > > open.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break
>> > change
>> > > > > which
>> > > > > > > > > should
>> > > > > > > > > > be
>> > > > > > > > > > > > > > > > highlighted
>> > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > > > > > FLIP,
>> > > > > > > > > > > > > > > > > > > > > > > > > may
>> > > > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility
>> part.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying
>> > asynchronous
>> > > > > > > > > > registration, I
>> > > > > > > > > > > > > would
>> > > > > > > > > > > > > > > > like
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > expect
>> > > > > > > > > > > > > > > > > > > > > > > > there
>> > > > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to
>> > open
>> > > > or
>> > > > > > > > register
>> > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > background,
>> > > > > > > > > > > > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > > > > > > > > > > > your previous
>> comments,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance
>> > will
>> > > be
>> > > > > > > > > initialized
>> > > > > > > > > > > if
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > > has
>> > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > been
>> > > > > > > > > > > > > > > > > > > > > > > > > initialized
>> > > > > > > > > > > > > > > > > > > > > > > > > > > yet. If the
>> > initialization
>> > > > > > process
>> > > > > > > > > fails,
>> > > > > > > > > > > > these
>> > > > > > > > > > > > > > > > > > statements
>> > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > > > > > executed
>> successfully."
>> > > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like
>> lazy
>> > > > > > > > initialization
>> > > > > > > > > > for
>> > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > than
>> > > > > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
>> > > > > > > > jinfeng1995@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
>> > > > > dev@flink.apache.org
>> > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月
>> 03日
>> > 下午
>> > > > > > 3:27:45
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP
>> > 295:
>> > > > > > Support
>> > > > > > > > > > > > persistence
>> > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > > > > configuration
>> > > > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
>> > > registration
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you
>> > all
>> > > > for
>> > > > > > your
>> > > > > > > > > > > interest
>> > > > > > > > > > > > in
>> > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > DISCUSS.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a
>> catalog
>> > > > with
>> > > > > > the
>> > > > > > > > same
>> > > > > > > > > > > name
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > both
>> > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > them?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a
>> > crucial
>> > > > > > point.
>> > > > > > > > > Based
>> > > > > > > > > > on
>> > > > > > > > > > > > my
>> > > > > > > > > > > > > > > > current
>> > > > > > > > > > > > > > > > > > > > personal
>> > > > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
>> > > > Map<String,
>> > > > > > > > Catalog>
>> > > > > > > > > > > > > catalogs
>> > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > serve
>> > > > > > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > > > > > > cache
>> > > > > > > > > > > > > > > > > > > > > > > > > > > for instantiated
>> catalogs
>> > > and
>> > > > > > have
>> > > > > > > > the
>> > > > > > > > > > > > highest
>> > > > > > > > > > > > > > > > > priority.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > There are three
>> methods
>> > > that
>> > > > > can
>> > > > > > > > affect
>> > > > > > > > > > the
>> > > > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs:
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1.
>> registerCatalog(String
>> > > > > > > > catalogName,
>> > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
>> > > catalog
>> > > > > > instance
>> > > > > > > > > > into
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2.
>> > unregisterCatalog(String
>> > > > > > > > > catalogName)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes
>> the
>> > > > catalog
>> > > > > > > > > instance
>> > > > > > > > > > > > > > > > corresponding
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > catalogName
>> > > > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
>> > > Catalog>
>> > > > > > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
>> > > > > catalogName)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method first
>> > retrieves
>> > > > the
>> > > > > > > > > > > corresponding
>> > > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > > > > > > > > > > > the Map<String,
>> Catalog>
>> > > > > > catalogs. If
>> > > > > > > > > the
>> > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > does
>> > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > > > exist,
>> > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > > > retrieves the
>> > corresponding
>> > > > > > > > > configuration
>> > > > > > > > > > > > from
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > CatalogStore,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and
>> puts
>> > > the
>> > > > > > > > > initialized
>> > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > > > > > into
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
>> > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > The following two
>> methods
>> > > > only
>> > > > > > modify
>> > > > > > > > > the
>> > > > > > > > > > > > > > > > configuration
>> > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
>> > > > > catalogName,
>> > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > String>
>> > > > > > > > > > > > > > > > > > > > > properties)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
>> > > > > properties
>> > > > > > to
>> > > > > > > > the
>> > > > > > > > > > > > > > > catalogStore
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > checks
>> > > > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
>> > > > catalogName
>> > > > > > with
>> > > > > > > > the
>> > > > > > > > > > > same
>> > > > > > > > > > > > > > name.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2.
>> removeCatalog(String
>> > > > > > catalogName)
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes
>> the
>> > > > > specified
>> > > > > > > > > > > > configuration
>> > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > specified
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
>> > > > > catalogStore.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > The following are
>> > possible
>> > > > > > conflict
>> > > > > > > > > > > > scenarios:
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When the
>> corresponding
>> > > > > > catalogName
>> > > > > > > > > > > already
>> > > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not
>> in
>> > the
>> > > > > > > > Map<String,
>> > > > > > > > > > > > > Catalog>,
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
>> > > > > > catalogName,
>> > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > > > > > succeed and be
>> directly
>> > > saved
>> > > > > to
>> > > > > > the
>> > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. When the
>> corresponding
>> > > > > > catalogName
>> > > > > > > > > > > already
>> > > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > both
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
>> > > > > Map<String,
>> > > > > > > > > > Catalog>,
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
>> > > catalog)
>> > > > > > method
>> > > > > > > > > will
>> > > > > > > > > > > > fail.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. When the
>> corresponding
>> > > > > > catalogName
>> > > > > > > > > > > already
>> > > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>,
>> the
>> > > > > > > > > > addCatalog(String
>> > > > > > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties)
>> > method
>> > > > can
>> > > > > > > > directly
>> > > > > > > > > > > save
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > properties
>> > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
>> > > > > > > > getCatalog(String
>> > > > > > > > > > > > > > > catalogName)
>> > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > > > > > > > > > use the new properties
>> > for
>> > > > > > > > > initialization
>> > > > > > > > > > > > > because
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > corresponding
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalog instance
>> already
>> > > > exists
>> > > > > > in
>> > > > > > > > > > catalogs
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > prioritized.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
>> > > > > > > > > > > unregisterCatalog(String
>> > > > > > > > > > > > > > > > > > catalogName)
>> > > > > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
>> > > > > > corresponding to
>> > > > > > > > > the
>> > > > > > > > > > > > > original
>> > > > > > > > > > > > > > > > > > > catalogName
>> > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > > > necessary.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it will
>> confuse
>> > > > users
>> > > > > > that
>> > > > > > > > > > > > > > > > > > > `registerCatalog(String
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
>> > > catalog)`
>> > > > > in
>> > > > > > the
>> > > > > > > > > > > > > `Map<String,
>> > > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > > catalogs`
>> > > > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> `registerCatalog(String
>> > > > > > catalogName,
>> > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > String>
>> > > > > > > > > > > > > > > > > > > > > > > properties)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This could potentially
>> > lead
>> > > > to
>> > > > > > > > > > confusion. I
>> > > > > > > > > > > > > > suggest
>> > > > > > > > > > > > > > > > > > > changing
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps
>> to
>> > > > > > > > > addCatalog(String
>> > > > > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties),
>> as
>> > > > > > previously
>> > > > > > > > > > > mentioned
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > @Hang
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > add
>> > > `registerCatalog(String
>> > > > > > > > > > > > > catalogName,Catalog
>> > > > > > > > > > > > > > > > > > catalog,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)`
>> method
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is
>> > already
>> > > an
>> > > > > > > > instance,
>> > > > > > > > > > > > adding
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > "lazyInit"
>> > > > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
>> > > > > > > > registerCatalog(String
>> > > > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > > > > > > > > > > method may not
>> > necessarily
>> > > > > > result in
>> > > > > > > > > lazy
>> > > > > > > > > > > > > > > > > initialization.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think
>> > about
>> > > > > > > > encryption
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
>> > > > > necessary,
>> > > > > > but
>> > > > > > > > > > > perhaps
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > encryption
>> > > > > > > > > > > > > > > > > > > > > logic
>> > > > > > > > > > > > > > > > > > > > > > > > > > > should be implemented
>> in
>> > > the
>> > > > > > > > > CatalogStore
>> > > > > > > > > > > > when
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > data
>> > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > actually
>> > > > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance,
>> the
>> > > > > > > > > FileCatalogStore
>> > > > > > > > > > > > could
>> > > > > > > > > > > > > > > > encrypt
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > data
>> > > > > > > > > > > > > > > > > > > > > > > when
>> > > > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
>> > > while
>> > > > > the
>> > > > > > > > > > > > > > MemoryCatalogStore
>> > > > > > > > > > > > > > > > > does
>> > > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > > > > > require
>> > > > > > > > > > > > > > > > > > > > > > > > > > > encryption.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need
>> the
>> > > > > > > > > > > > `MemoryCatalogStore`?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it is
>> necessary
>> > to
>> > > > > have a
>> > > > > > > > > > > > > > > MemoryCatalogStore
>> > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > default
>> > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, which
>> > saves
>> > > > > > Catalog
>> > > > > > > > > > > > > > configurations
>> > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > memory.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want
>> to
>> > > > > > implement
>> > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > loading
>> > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > Catalog,
>> > > > > > > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > > > > > > > would need to
>> introduce
>> > > > > > additional
>> > > > > > > > > cache.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
>> > > > > > registration
>> > > > > > > > > > failed?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This is also a
>> critical
>> > > > > concern.
>> > > > > > > > When
>> > > > > > > > > > > > > executing
>> > > > > > > > > > > > > > > DDL,
>> > > > > > > > > > > > > > > > > > DQL,
>> > > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > DML
>> > > > > > > > > > > > > > > > > > > > > > > > > > > statements that
>> > reference a
>> > > > > > specific
>> > > > > > > > > > > Catalog,
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > > > > > > > > > > > > will be initialized
>> if it
>> > > has
>> > > > > not
>> > > > > > > > been
>> > > > > > > > > > > > > > initialized
>> > > > > > > > > > > > > > > > yet.
>> > > > > > > > > > > > > > > > > > If
>> > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > initialization process
>> > > fails,
>> > > > > > these
>> > > > > > > > > > > > statements
>> > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > executed
>> > > > > > > > > > > > > > > > > > > > > > > > > > > successfully.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
>> > > > Catalog>
>> > > > > > > > catalogs
>> > > > > > > > > > and
>> > > > > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > > > > > > catalogstore
>> > > > > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all
>> > have
>> > > > the
>> > > > > > same
>> > > > > > > > > > > catalog
>> > > > > > > > > > > > > > name,
>> > > > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > > > correponding
>> > > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
>> > > instances.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This issue can be
>> > resolved
>> > > by
>> > > > > > > > referring
>> > > > > > > > > > to
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > previous
>> > > > > > > > > > > > > > > > > > > > > > responses.
>> > > > > > > > > > > > > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > > > > > > > > > > > key principle is that
>> the
>> > > > > Catalog
>> > > > > > > > that
>> > > > > > > > > > has
>> > > > > > > > > > > > been
>> > > > > > > > > > > > > > > most
>> > > > > > > > > > > > > > > > > > > recently
>> > > > > > > > > > > > > > > > > > > > > > used
>> > > > > > > > > > > > > > > > > > > > > > > > > > > will be given priority
>> > for
>> > > > > > subsequent
>> > > > > > > > > > use.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
>> > > > 10:58 PM
>> > > > > > > > Xianxun
>> > > > > > > > > > Ye
>> > > > > > > > > > > <
>> > > > > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving
>> this
>> > > > Flip,
>> > > > > > I do
>> > > > > > > > > > > believe
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > Flip
>> > > > > > > > > > > > > > > > > > > could
>> > > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > > helpful
>> > > > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I
>> > work
>> > > > for
>> > > > > > also
>> > > > > > > > > > > manages a
>> > > > > > > > > > > > > lot
>> > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > catalogs,
>> > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks
>> > becomes
>> > > > > slow
>> > > > > > > > > because
>> > > > > > > > > > of
>> > > > > > > > > > > > > > > loading a
>> > > > > > > > > > > > > > > > > > > number
>> > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > We obtain the
>> catalogs in
>> > > the
>> > > > > > user's
>> > > > > > > > > > Flink
>> > > > > > > > > > > > SQL
>> > > > > > > > > > > > > > > > through
>> > > > > > > > > > > > > > > > > > > > regular
>> > > > > > > > > > > > > > > > > > > > > > > > > expression
>> > > > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
>> > > catalogs
>> > > > > to
>> > > > > > > > > improve
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > speed
>> > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > task
>> > > > > > > > > > > > > > > > > > > > > > > > submission.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > After reading
>> flip-295, I
>> > > > have
>> > > > > > some
>> > > > > > > > > > > > questions:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
>> > > catalog
>> > > > > with
>> > > > > > > > > CREATE
>> > > > > > > > > > > > > CATALOG,
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
>> > used
>> > > > by
>> > > > > > > > default.
>> > > > > > > > > > > What
>> > > > > > > > > > > > if
>> > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > registration
>> > > > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
>> > > implement
>> > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > registration?
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the
>> same
>> > > > > > question
>> > > > > > > > with
>> > > > > > > > > > > > > Shammon’s
>> > > > > > > > > > > > > > > > first
>> > > > > > > > > > > > > > > > > > > > > comment.
>> > > > > > > > > > > > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
>> > > catalogs
>> > > > > and
>> > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > catalogstore
>> > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all
>> have
>> > the
>> > > > > same
>> > > > > > > > > catalog
>> > > > > > > > > > > > name,
>> > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > correponding
>> > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
>> > > instances.
>> > > > > For
>> > > > > > > > > example:
>> > > > > > > > > > > > > > > catalog1->
>> > > > > > > > > > > > > > > > > > > > > jdbcCatalog
>> > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but
>> catalog1 ->
>> > > > > > hiveCatalog
>> > > > > > > > > in
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > catalogstore.
>> > > > > > > > > > > > > > > > > > > > > Would
>> > > > > > > > > > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > > > > > > > > case
>> > > > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we
>> should
>> > > > define
>> > > > > > > > > `catalogs`
>> > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > `catalogstore`
>> > > > > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > > > > > > > clearly.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日
>> 11:13，Hang
>> > > > Ruan
>> > > > > <
>> > > > > > > > > > > > > > > > ruanhang1993@gmail.com
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > 写道：
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Samrat,
Thanks for your advice.
> 1. The createCatalog method does not mention any exceptions being thrown.
CreateCatalog will throw CatalogException like registerCatalog. As
CatalogException is a RuntimeException,
there is no explicit declaration of throwing Exceptions in CatalogManager
and TableEnvironment.
To avoid misunderstandings, I have added the "throw CatalogException" flag
to the createCatalog method definition of CatalogStore.
> 2. Could you please provide an exhaustive list of the supported kinds?
Sure, the documentation now includes the configuration of the built-in
CatalogStore as well as how to configure a custom CatalogStore.
Best,
Feng
On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com> wrote:
> Hi Feng,
>
> Thank you for providing the proposal. I believe this feature will be highly
> valuable.
>
> I have a couple of inquiries:
>
> 1. According to the documentation [1], the createCatalog method does not
> mention any exceptions being thrown. However, I would like to confirm if it
> is always true that there will be no failures in all scenarios. Please let
> me know if there is any additional information I may have missed.
>
> 2. Regarding the registration process using the `table.catalog-store.kind`
> configuration, could you please provide an exhaustive list of the supported
> kinds?
> It would be great to have a comprehensive understanding of the options
> available.
>
> Bests,
> Samrat
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>
> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
>
> > Hi Hang and Jark
> >
> > Thank you very much for your reply.
> >
> > @Jark
> > > 1. Could you move the CatalogStore registration API to the "Public
> > Interface" section?
> >
> > Indeed, it is more reasonable.
> >
> > > 2. We should prefix "table." for the CatalogStore configuration.
> >
> > Sure, the names that you provided are indeed more logical and easier to
> > understand.
> >
> > > 3. About the open/close method in CatalogStoreFactory.
> >
> > I also agree with the proposed requirement scenario and design. I have
> > already made modifications to the interface.
> >
> > @Hang
> >
> > > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> > the resource.
> >
> > I have already added the missing method.
> >
> > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >
> > Sorry for the misspelling, I updated this part.
> >
> > > 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
> > to init or close its resource
> >
> > This is indeed a scenario that needs to be considered, and I have added
> > relevant methods accordingly.
> > Additionally, I have included some relevant description information in
> the
> > documentation to help others understand
> > why the open/close methods need to be added.
> >
> >
> > Best,
> > Feng
> >
> > On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> >
> > > Hi Jing,
> > >
> > > Thank you for the update.
> > >
> > > 1. Could you move the CatalogStore registration API to the "Public
> > > Interface" section?
> > > "Proposed Changes" is more like a place to describe the implementation
> > > details.
> > >
> > > 2. We should prefix "table." for the CatalogStore configuration.
> Besides,
> > > the config key
> > > name should be hierarchical[1]. Therefore, it may be better to use:
> > > "table.catalog-store.kind"
> > > "table.catalog-store.file.path"
> > >
> > > 3. I think Hang's suggestions make sense.
> > >
> > > Others look good to me.
> > >
> > > Best,
> > > Jark
> > >
> > > On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com> wrote:
> > >
> > > > Hi, Feng.
> > > >
> > > > Thanks for the update.
> > > > The current design looks good to me. I have some minor comments.
> > > >
> > > > 1. The `CatalogStore` need the `open`/`close` methods to init or
> close
> > > the
> > > > resource. For example, when storing the information in MySQL, the
> store
> > > > needs to open and close the connections.
> > > >
> > > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > > >
> > > > 3. About the usage in the sql gateway.
> > > > Considering the usage in sql gateway, the sql gateway may create a
> > > > CatalogStore for each session.
> > > > If we are using the MySqlCatalogStore, there would be so many
> > > connections.
> > > > How can we reuse the connection among these sessions?
> > > > I think sql gateway need to maintain a connection pool in
> > > > the CatalogStoreFactory and each session get its own connection from
> > the
> > > > pool when it is created.
> > > > Then the `CatalogStoreFactory` may need the `open`/`close` methods to
> > > init
> > > > or close its resource.
> > > > Or is there a better way?
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> > > >
> > > > > Thanks Jingsong.
> > > > >
> > > > > > Just naming, maybe `createCatalog` in TableEnv
> > > > >
> > > > > +1 For this, I have already updated FLIP.
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > >
> > > > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> jingsonglee0@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Feng,
> > > > > >
> > > > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > > > > functions are converted to createxxx from registerxxx.
> > > > > >
> > > > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > Hi jark, thanks for your suggestion.
> > > > > > >
> > > > > > > > 1. How to register the CatalogStore for Table API? I think
> the
> > > > > > > CatalogStore should be immutable once TableEnv is created.
> > > Otherwise,
> > > > > > there
> > > > > > > might be some data inconsistencies when CatalogStore is
> changed.
> > > > > > >
> > > > > > > Yes, We should initialize the CatalogStore when creating the
> > > > TableEnv.
> > > > > > > Therefore, my current proposal is to add a method to configure
> > the
> > > > > > > CatalogStore in EnvironmentSettings.
> > > > > > >
> > > > > > > // Initialize a catalog Store
> > > > > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > > > > >
> > > > > > > // set up the Table API
> > > > > > > final EnvironmentSettings settings =
> > > > > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > > > > .withCatalogStore(catalogStore)
> > > > > > > .build();
> > > > > > >
> > > > > > > final TableEnvironment tableEnv =
> > > > > > TableEnvironment.create(settings);
> > > > > > >
> > > > > > >
> > > > > > > > 2. Why does the CatalogStoreFactory interface only have a
> > default
> > > > > > method,
> > > > > > > not an interface method?
> > > > > > >
> > > > > > > Sorry, While I did refer to the Catalog interface, I agree that
> > as
> > > a
> > > > > new
> > > > > > > interface, the CatalogStoreFactory should not have a default
> > method
> > > > but
> > > > > > an
> > > > > > > interface method. I have already modified the interface.
> > > > > > >
> > > > > > > > 3. Please mention the alternative API in Javadoc for the
> > > > > > > deprecated`registerCatalog`.
> > > > > > > > 4. In the "Compatibility" section, would be better to mention
> > the
> > > > > > changed
> > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> > other
> > > > > > > persisted catalog store) is used.
> > > > > > >
> > > > > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > > > > >
> > > > > > >
> > > > > > > [1].
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
> wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > > > > The current design looks pretty good to me. I just have some
> > > minor
> > > > > > > > comments.
> > > > > > > >
> > > > > > > > 1. How to register the CatalogStore for Table API? I think
> the
> > > > > > CatalogStore
> > > > > > > > should be immutable once TableEnv is created. Otherwise,
> there
> > > > might
> > > > > be
> > > > > > > > some data inconsistencies when CatalogStore is changed.
> > > > > > > >
> > > > > > > > 2. Why does the CatalogStoreFactory interface only have a
> > default
> > > > > > method,
> > > > > > > > not an interface method?
> > > > > > > >
> > > > > > > > 3. Please mention the alternative API in Javadoc for the
> > > deprecated
> > > > > > > > `registerCatalog`.
> > > > > > > >
> > > > > > > > 4. In the "Compatibility" section, would be better to mention
> > the
> > > > > > changed
> > > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> > > other
> > > > > > > > persisted catalog store) is used.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jark
> > > > > > > >
> > > > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <jinfeng1995@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi ， thanks all for reviewing the flip.
> > > > > > > > >
> > > > > > > > > @Ron
> > > > > > > > >
> > > > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore
> > method,
> > > > do
> > > > > we
> > > > > > > > need
> > > > > > > > > to provide a default implementation?
> > > > > > > > >
> > > > > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory
> to
> > > > > create
> > > > > > an
> > > > > > > > > InMemoryCatalogStore.
> > > > > > > > >
> > > > > > > > > > If we get a Catalog from CatalogStore, after
> initializing
> > > it,
> > > > > > whether
> > > > > > > > we
> > > > > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > > > > >
> > > > > > > > > Yes, in the current design, catalogs are stored as
> snapshots,
> > > and
> > > > > > once
> > > > > > > > > initialized, the Catalog will be placed in the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs.
> > > > > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> > > > primary
> > > > > > > > source
> > > > > > > > > for obtaining the corresponding Catalog.
> > > > > > > > >
> > > > > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > > > > `catalog.store.path`?
> > > > > > > > >
> > > > > > > > > I think it is okay. Adding "sql" at the beginning may seem
> a
> > > bit
> > > > > > > > strange. I
> > > > > > > > > will update the FLIP.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > @Shammon
> > > > > > > > >
> > > > > > > > > Thank you for the review. I have made the necessary
> > > corrections.
> > > > > > > > > Regarding the modifications made to the Public Interface, I
> > > have
> > > > > also
> > > > > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> > zjureel@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for updating, I have some minor comments
> > > > > > > > > >
> > > > > > > > > > 1. The modification of `CatalogManager` should not be in
> > > > `Public
> > > > > > > > > > Interfaces`, it is not a public interface.
> > > > > > > > > >
> > > > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore`
> and
> > > > > > > > > > `CatalogStoreFactory`
> > > > > > > > > >
> > > > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
> > > should
> > > > be
> > > > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > > > > catalogStore.get(catalogName);`
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shammon FY
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <
> > ron9.liu@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Feng
> > > > > > > > > > >
> > > > > > > > > > > Thanks for driving this FLIP, this proposal is very
> > useful
> > > > for
> > > > > > > > catalog
> > > > > > > > > > > management.
> > > > > > > > > > > I have some small questions:
> > > > > > > > > > >
> > > > > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> > > > method,
> > > > > > do we
> > > > > > > > > > need
> > > > > > > > > > > to provide a default implementation?
> > > > > > > > > > > 2. If we get Catalog from CatalogStore, after
> > initializing
> > > > it,
> > > > > > > > whether
> > > > > > > > > we
> > > > > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > > > > `sql.catalog.store.file.path`, how about renaming them
> to
> > > > > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Ron
> > > > > > > > > > >
> > > > > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一
> 21:19写道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi yuxia
> > > > > > > > > > > >
> > > > > > > > > > > > > But from the code in Proposed Changes, once we
> > > register
> > > > > the
> > > > > > > > > Catalog,
> > > > > > > > > > > we
> > > > > > > > > > > > initialize it and open it. right?
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> > > > original
> > > > > > > > CREATE
> > > > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > > > > registerCatalog
> > > > > > > > > so
> > > > > > > > > > > > that parameter validation can be performed.
> > > > > > > > > > > >
> > > > > > > > > > > > In the current design, lazy initialization is mainly
> > > > > reflected
> > > > > > in
> > > > > > > > > > > > getCatalog. If CatalogStore has already saved some
> > > catalog
> > > > > > > > > > > configurations,
> > > > > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > > > > luoyuxia@alumni.sjtu.edu.cn
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi, Feng.
> > > > > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > > > > initialization*.
> > > > > > > > > If
> > > > > > > > > > > i'm
> > > > > > > > > > > > > wrong, please correct me.
> > > > > > > > > > > > >
> > > > > > > > > > > > > IIUC, lazy initialization means only you need to
> > access
> > > > the
> > > > > > > > > catalog,
> > > > > > > > > > > then
> > > > > > > > > > > > > you initialize it. But from the code in Proposed
> > > Changes,
> > > > > > once we
> > > > > > > > > > > > register
> > > > > > > > > > > > > the Catalog,
> > > > > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > >
> > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > > > Catalog
> > > > > > > > > > > configuration
> > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One of the major changes is that current design
> will
> > > > > provide
> > > > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead
> > of a
> > > > > > cache,
> > > > > > > > > which
> > > > > > > > > > > > means
> > > > > > > > > > > > > once it has been initialized, any changes done by
> > other
> > > > > > sessions
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > affect it. Point 6 described follow-up options for
> > > > further
> > > > > > > > > > improvement.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all, I would like to update you on the latest
> > > > progress
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon
> > FY,
> > > > > > ShengKai
> > > > > > > > > Fang
> > > > > > > > > > > > and I
> > > > > > > > > > > > > > had an offline discussion regarding the overall
> > > > solution
> > > > > > for
> > > > > > > > > Flink
> > > > > > > > > > > > > > CatalogStore. We have reached a consensus and I
> > have
> > > > > > updated
> > > > > > > > the
> > > > > > > > > > > final
> > > > > > > > > > > > > > solution in FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > > > > configuration
> > > > > > > > > > similar
> > > > > > > > > > > > to
> > > > > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > > > > storeCatalog(String
> > > > > > > > > > > > > > catalogName, CatalogDescriptor) and
> > > > CatalogDescriptor
> > > > > > > > > > > > > getCatalog(String)
> > > > > > > > > > > > > > will both use CatalogDescriptor instead of
> > Catalog
> > > > > > instance.
> > > > > > > > > > This
> > > > > > > > > > > > way,
> > > > > > > > > > > > > > CatalogStore will only be responsible for
> saving
> > > and
> > > > > > > > > retrieving
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > configurations without having to initialize
> > > > catalogs.
> > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The default registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > > > > function in CatalogManager will be marked as
> > > > > deprecated.
> > > > > > > > > > > > > > 4.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > A new function registerCatalog(String
> > catalogName,
> > > > > > > > > > > CatalogDescriptor
> > > > > > > > > > > > > > catalog) will be added to serve as the default
> > > > > > registration
> > > > > > > > > > > function
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > > > > 5.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager
> > will
> > > > > > remain
> > > > > > > > > > > unchanged
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > save initialized catalogs.This means that
> > deletion
> > > > > > > > operations
> > > > > > > > > > from
> > > > > > > > > > > > one
> > > > > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > > > > 6.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > To support multi-session synchronization
> > scenarios
> > > > for
> > > > > > > > > deletions
> > > > > > > > > > > > later
> > > > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > > > > configurable.There
> > > > > > > > > > > may
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > three possible situations:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > > > > automatically
> > > > > > > > > clear
> > > > > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > c.No caching of any instances; each call to
> > > > getCatalog
> > > > > > > > > creates a
> > > > > > > > > > > new
> > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you very much for your attention and
> > > suggestions
> > > > on
> > > > > > this
> > > > > > > > > > > FLIP. A
> > > > > > > > > > > > > > special thanks to Hang Ruan for his suggestions
> on
> > > the
> > > > > > entire
> > > > > > > > > > design
> > > > > > > > > > > > and
> > > > > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If you have any further suggestions or feedback
> > about
> > > > > this
> > > > > > FLIP
> > > > > > > > > > > please
> > > > > > > > > > > > > feel
> > > > > > > > > > > > > > free to share.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for improving the FLIP. It looks good to
> > me.
> > > > We
> > > > > > could
> > > > > > > > > > still
> > > > > > > > > > > > > > > reconsider in the future how to provide more
> > common
> > > > > > built-in
> > > > > > > > > > cache
> > > > > > > > > > > > > > > functionality in CatalogManager so that not
> every
> > > > > > > > CatalogSotre
> > > > > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There might be more such issues. I would
> > > suggest
> > > > > you
> > > > > > > > > > > completely
> > > > > > > > > > > > > walk
> > > > > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I am very sorry for my carelessness and at
> the
> > > same
> > > > > > time, I
> > > > > > > > > > > greatly
> > > > > > > > > > > > > > > > appreciate your careful review. I have
> > thoroughly
> > > > > > checked
> > > > > > > > the
> > > > > > > > > > > > entire
> > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If I am not mistaken, with the current
> FLIP
> > > > > design,
> > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > could work without Optional CatalogStore
> being
> > > > > > configured.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Yes, in the original design, CatalogStore was
> > not
> > > > > > necessary
> > > > > > > > > > > because
> > > > > > > > > > > > > > > > CatalogManager used Map<String, Catalog>
> > catalogs
> > > > to
> > > > > > store
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > However, this caused inconsistency issues.
> > > > > Therefore, I
> > > > > > > > > > modified
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > of the design and removed Map<String,
> Catalog>
> > > > > catalogs
> > > > > > > > from
> > > > > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > > > > At the same time, InMemoryCatalog will serve
> > as
> > > > the
> > > > > > > > default
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > to save catalogs in memory and replace the
> > > > > > functionality of
> > > > > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > > > > The previous plan that kept
> > > > > > Map<String,Catalog>catalogs has
> > > > > > > > > > been
> > > > > > > > > > > > > moved
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are still many places contain
> > > inconsistent
> > > > > > content,
> > > > > > > > > > e.g.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. "Asynchronous registration" is still
> used.
> > > > > > > > > > > > > > > > > 2. The java comment of the method
> > > > > > registerCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not
> > > tell
> > > > > > what the
> > > > > > > > > > > method
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > doing.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There might be more such issues. I would
> > > suggest
> > > > > you
> > > > > > > > > > completely
> > > > > > > > > > > > > walk
> > > > > > > > > > > > > > > > > through the FLIP again and fix those
> issues.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean
> > > that
> > > > > you
> > > > > > will
> > > > > > > > > > build
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > functionality in the CatalogStore? This is
> a
> > > very
> > > > > > > > different
> > > > > > > > > > > > design
> > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > from what the current FLIP described. If I
> am
> > > not
> > > > > > > > mistaken,
> > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > current FLIP design, CatalogManager could
> > work
> > > > > > without
> > > > > > > > > > Optional
> > > > > > > > > > > > > > > > > CatalogStore being configured. That is the
> > > reason
> > > > > > why I
> > > > > > > > > > > mentioned
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > last email that the example code wrt the
> > > Optional
> > > > > > > > > > CatalogStore
> > > > > > > > > > > is
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There are still some NIT issues in the
> > FLIP
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you very much for the careful
> > review. I
> > > > > have
> > > > > > > > > already
> > > > > > > > > > > made
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > > > multi-instance
> > > > > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Currently, there are conflicts in
> multiple
> > > > > > scenarios
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > design. I am thinking whether we should
> > > remove
> > > > > > > > > 'Map<String,
> > > > > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > > > > InMemoryCatalogStore.
> > > > > > > > > > This
> > > > > > > > > > > > > way,
> > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > can implement their own CatalogStore to
> > > achieve
> > > > > > > > > > > multi-instance
> > > > > > > > > > > > > > > > > > non-conflicting scenarios. What do you
> > think?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There
> are
> > > > still
> > > > > > some
> > > > > > > > > NIT
> > > > > > > > > > > > issues
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore
> > has
> > > > been
> > > > > > used
> > > > > > > > as
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > instead of Optional in the code
> example.
> > It
> > > > > > should be
> > > > > > > > > > fine
> > > > > > > > > > > to
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > pseudo code for now and update it after
> > you
> > > > > > submit
> > > > > > > > the
> > > > > > > > > > PR.
> > > > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used
> > somewhere
> > > in
> > > > > the
> > > > > > > > > > rejected
> > > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > should be persistContext(...) to keep
> it
> > > > > > consistent.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > multi-instance
> > > > > > > > > > > > > scenarios,
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > sure if this is the intended behaviour.
> > If
> > > > > > > > Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > used as a cache, it should be invalid,
> > once
> > > > the
> > > > > > > > related
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > removed from the CatalogStore by
> another
> > > > > > instance.
> > > > > > > > Did
> > > > > > > > > I
> > > > > > > > > > > miss
> > > > > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng
> Jin
> > <
> > > > > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > > > > I think this is a good function
> name, I
> > > > have
> > > > > > > > updated
> > > > > > > > > it
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Some common cache features
> should
> > be
> > > > > > > > implemented
> > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> > > > alternative
> > > > > 1
> > > > > > > > turns
> > > > > > > > > > out
> > > > > > > > > > > to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > appropriate later, I will improve
> this
> > > part
> > > > > of
> > > > > > the
> > > > > > > > > > > content.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > As the above discussion moves
> > forward,
> > > > the
> > > > > > > > option 2
> > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
> > > offline,
> > > > > we
> > > > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > > > > inconsistency
> > > > > > > > > > issues.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There are some inconsistent
> > > descriptions
> > > > in
> > > > > > the
> > > > > > > > > > > content.
> > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > > > > I will do my best to improve the
> > document
> > > > and
> > > > > > > > > > appreciate
> > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > can you put the unselected option
> in
> > > > > > `Rejected
> > > > > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > > > > Sure, I have moved it to the
> `Rejected
> > > > > > > > Alternatives`.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM
> Shammon
> > > FY
> > > > <
> > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I found there are two options in
> > > > `Proposed
> > > > > > > > > Changes`,
> > > > > > > > > > > can
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > put
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > > > > Alternatives`? I
> > > > > > > > > think
> > > > > > > > > > > > this
> > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM
> Jing
> > Ge
> > > > > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I
> am
> > > > still
> > > > > > > > confused
> > > > > > > > > > > after
> > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > > the thread with following
> > questions:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
> > > registerCatalog()
> > > > > and
> > > > > > > > > > > > addCatalog()
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > > > > difference based on their names.
> > One
> > > of
> > > > > > them is
> > > > > > > > > > > > > responsible
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > > persistence. How about
> > > > persistCatalog()?
> > > > > > > > > > > > > > > > > > > > > > 2. As you mentioned that
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > > > > persistence.
> > > > > > > > I
> > > > > > > > > > > would
> > > > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > > > > their purpose conceptually and
> > > clearly
> > > > in
> > > > > > the
> > > > > > > > > FLIP.
> > > > > > > > > > > > Some
> > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > features should be implemented,
> > i.e.
> > > > data
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > cache
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog
> > instance
> > > > > > should be
> > > > > > > > > > found
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the cache(either it has been
> > > > initialized
> > > > > > or it
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > > > > consistency
> > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > taken
> > > > > > > > > > > > > > care
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> > > > forward,
> > > > > > the
> > > > > > > > > > option
> > > > > > > > > > > 2
> > > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> > > > because,
> > > > > > afaiu,
> > > > > > > > > > > issues
> > > > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > previously with option 1 are not
> > > solved
> > > > > > yet. Do
> > > > > > > > > you
> > > > > > > > > > > > still
> > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > > > > both options and ask for
> > suggestions
> > > > for
> > > > > > both
> > > > > > > > of
> > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP,
> > there
> > > > are
> > > > > > some
> > > > > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the content. Would you like to
> > clean
> > > > > them
> > > > > > up?
> > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM
> Feng
> > > > Jin <
> > > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thank you for your response,
> and
> > I
> > > > > > completely
> > > > > > > > > > agree
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > > > > Initially, I may have over
> > > > complicated
> > > > > > the
> > > > > > > > > whole
> > > > > > > > > > > > issue.
> > > > > > > > > > > > > > > First
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > > > > we need to consider the
> > persistence
> > > > of
> > > > > > the
> > > > > > > > > > > Catalog's
> > > > > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > > > > If we only need to provide
> > > > persistence
> > > > > > for
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > add a toConfiguration method to
> > the
> > > > > > Catalog
> > > > > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > > > > This method can convert a
> Catalog
> > > > > > instance
> > > > > > > > to a
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > > > properties, and the default
> > > > > > implementation
> > > > > > > > will
> > > > > > > > > > > throw
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > > > > * Returns a map containing
> the
> > > > > > properties
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > > > > * @return a map containing
> the
> > > > > > properties
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > > > > * @throws
> > > > > > UnsupportedOperationException if
> > > > > > > > > the
> > > > > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > > > > * the default implementation
> > of
> > > > this
> > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > > > > toProperties()
> > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The specific process is as
> > follows:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. If the user has configured
> a
> > > > > > > > CatalogStore,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > > will be called when
> registering a
> > > > > Catalog
> > > > > > > > > > instance
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > > catalog).
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > will be converted to a
> > Map<String,
> > > > > > String>
> > > > > > > > > > > properties
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> > > > instances
> > > > > > do
> > > > > > > > not
> > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > > > > CatalogStore,
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > > > > consistency
> > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > > original
> > > > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > > > the same time can also avoid
> > > > conflicts
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
> > > itself
> > > > > and
> > > > > > > > > provide
> > > > > > > > > > a
> > > > > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > implementation for lazy
> loading,
> > > such
> > > > > as
> > > > > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > public class
> ConfigurationCatalog
> > > > > > implements
> > > > > > > > > > > Catalog
> > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > >
> ConfigurationCatalog(Map<String,
> > > > > String>
> > > > > > > > > > > properties)
> > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I have added this design to the
> > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> > > > Shammon
> > > > > > FY <
> > > > > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > > > > functionality
> > > > > > > > > of
> > > > > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > different operations of
> > multiple
> > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> > > > implementation
> > > > > in
> > > > > > > > Trino
> > > > > > > > > > is
> > > > > > > > > > > > also
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino
> only
> > > has
> > > > > one
> > > > > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > catalog to memory and
> > > > `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> > > > another
> > > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > save a catalog in
> `Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you
> > mentioned
> > > > > > before,
> > > > > > > > the
> > > > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > > > > `CatalogManager`.
> > > > > > > > How
> > > > > > > > > > > about
> > > > > > > > > > > > > > > saving
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog>
> catalogs`
> > > and
> > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > > > > when it is registered or
> added
> > in
> > > > > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> 5:22 PM
> > > Feng
> > > > > > Jin <
> > > > > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I
> > am
> > > > very
> > > > > > sorry
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
> > > original
> > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > > > pre-discussion
> > > > > > [1]
> > > > > > > > for
> > > > > > > > > > > this
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such
> a
> > > > > > discussion
> > > > > > > > > > before.
> > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > whole solution, I found
> that
> > > the
> > > > > > logic of
> > > > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > > > need to change much. *We
> > cannot
> > > > > only
> > > > > > > > > persist
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > > > > registerCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > > > > not be enough to save
> > Catalogs,
> > > > > > because
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > end
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > > > the configurations
> > > corresponding
> > > > to
> > > > > > the
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > > > > CatalogStore
> > > > > > > > > > interface
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> > > > part, I
> > > > > > also
> > > > > > > > > took
> > > > > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> > handling
> > > > > > catalog
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the
> > suggestion. I
> > > > > will
> > > > > > > > > provide
> > > > > > > > > > a
> > > > > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > code examples for this
> part,
> > > and
> > > > > add
> > > > > > it
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > > > `addCatalog`
> > > > > > > > > still
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > > catalog
> > > > > to
> > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> very
> > > > > personal
> > > > > > > > > > opinion,
> > > > > > > > > > > > you
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
> > > intuitive
> > > > > to
> > > > > > me.
> > > > > > > > I
> > > > > > > > > > > don't
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
> > > statement,
> > > > > > which
> > > > > > > > > > method
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > `addCatalog`?
> > > > > > I'm
> > > > > > > > > > > wondering
> > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my
> > original
> > > > > > design
> > > > > > > > was
> > > > > > > > > to
> > > > > > > > > > > add
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this
> would
> > > > > disrupt
> > > > > > the
> > > > > > > > > > > existing
> > > > > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > > > > configuration to
> > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the
> > > same
> > > > > time
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really
> neccessary
> > to
> > > > > > provide a
> > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> > > > otherwise
> > > > > we
> > > > > > > > would
> > > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > > > > Map to store the
> > configuration
> > > > for
> > > > > > lazy
> > > > > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > > registration
> > > > > > for
> > > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > > I don't think
> > > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> > > > interface
> > > > > > > > because
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > initialization
> > > > > > > > > for
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > > > > inaccurate. It
> > > > > > > > > > > should
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > > > > registration. I
> > > > > > > > > have
> > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> > 4:27 PM
> > > > > > Shammon
> > > > > > > > FY <
> > > > > > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > > > > `registerCatalog`
> > > > > > > > > > > method
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
> > > method
> > > > > > named
> > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> > > > memory
> > > > > > > > catalog
> > > > > > > > > > is a
> > > > > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> > > > exposed
> > > > > to
> > > > > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > > > pre-discussion
> > > > > > [1]
> > > > > > > > for
> > > > > > > > > > > this
> > > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
> > > conclusion
> > > > > of
> > > > > > that
> > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> > > > interface
> > > > > > and
> > > > > > > > > > > implement
> > > > > > > > > > > > it
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > > > as memory, file and
> > external
> > > > > > system.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
> > > between
> > > > > the
> > > > > > > > > current
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > > > > What about the proposal
> of
> > > the
> > > > > > > > discussion
> > > > > > > > > > in
> > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > >
> > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> > > 3:41 PM
> > > > > > yuxia <
> > > > > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving
> > > this
> > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> > > handling
> > > > > > catalog
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
> > > it'll
> > > > be
> > > > > > > > better
> > > > > > > > > to
> > > > > > > > > > > > > explain
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > these methods and give
> a
> > > > brief
> > > > > > > > example
> > > > > > > > > in
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > > > `addCatalog`
> > > > > > > > > > still
> > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > > > this method is for
> > storing
> > > > > > catalog to
> > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> > > very
> > > > > > personal
> > > > > > > > > > > > opinion,
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > > > > statement,
> > > > > > > > which
> > > > > > > > > > > method
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > > > `addCatalog`?
> > > > > > > > I'm
> > > > > > > > > > > > > wondering
> > > > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with
> SQL
> > > > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really
> > neccessary
> > > to
> > > > > > > > provide a
> > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > > > registration
> > > > > > > > for
> > > > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > When creating a
> catalog
> > > > with
> > > > > > CREATE
> > > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
> > used
> > > > by
> > > > > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
> > > registration
> > > > is
> > > > > > the
> > > > > > > > > > default
> > > > > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > > > > synchronous
> > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > > > > Will both method
> > > `addCatalog`
> > > > > and
> > > > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > > > > registration,
> > > > > > > > it
> > > > > > > > > > may
> > > > > > > > > > > > well
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > > > > executes successfully,
> > but
> > > > then
> > > > > > the
> > > > > > > > > > > following
> > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog
> fail
> > > to
> > > > > > open.
> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break
> > change
> > > > > which
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility
> part.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying
> > asynchronous
> > > > > > > > > > registration, I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to
> > open
> > > > or
> > > > > > > > register
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance
> > will
> > > be
> > > > > > > > > initialized
> > > > > > > > > > > if
> > > > > > > > > > > > it
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > > > > yet. If the
> > initialization
> > > > > > process
> > > > > > > > > fails,
> > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > > > initialization
> > > > > > > > > > for
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > > > > dev@flink.apache.org
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月
> 03日
> > 下午
> > > > > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP
> > 295:
> > > > > > Support
> > > > > > > > > > > > persistence
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
> > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you
> > all
> > > > for
> > > > > > your
> > > > > > > > > > > interest
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a
> catalog
> > > > with
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > name
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a
> > crucial
> > > > > > point.
> > > > > > > > > Based
> > > > > > > > > > on
> > > > > > > > > > > > my
> > > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> > > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > > > > for instantiated
> catalogs
> > > and
> > > > > > have
> > > > > > > > the
> > > > > > > > > > > > highest
> > > > > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > There are three methods
> > > that
> > > > > can
> > > > > > > > affect
> > > > > > > > > > the
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1.
> registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
> > > catalog
> > > > > > instance
> > > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2.
> > unregisterCatalog(String
> > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > > catalog
> > > > > > > > > instance
> > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
> > > Catalog>
> > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method first
> > retrieves
> > > > the
> > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > > the Map<String,
> Catalog>
> > > > > > catalogs. If
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > > retrieves the
> > corresponding
> > > > > > > > > configuration
> > > > > > > > > > > > from
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and
> puts
> > > the
> > > > > > > > > initialized
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The following two
> methods
> > > > only
> > > > > > modify
> > > > > > > > > the
> > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > > > > catalogName,
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > > > > properties
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> > > > catalogName
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > > > specified
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > > > > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The following are
> > possible
> > > > > > conflict
> > > > > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When the
> corresponding
> > > > > > catalogName
> > > > > > > > > > > already
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in
> > the
> > > > > > > > Map<String,
> > > > > > > > > > > > > Catalog>,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly
> > > saved
> > > > > to
> > > > > > the
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. When the
> corresponding
> > > > > > catalogName
> > > > > > > > > > > already
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > > > > Map<String,
> > > > > > > > > > Catalog>,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
> > > catalog)
> > > > > > method
> > > > > > > > > will
> > > > > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. When the
> corresponding
> > > > > > catalogName
> > > > > > > > > > > already
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>,
> the
> > > > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties)
> > method
> > > > can
> > > > > > > > directly
> > > > > > > > > > > save
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > > > > getCatalog(String
> > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > > > use the new properties
> > for
> > > > > > > > > initialization
> > > > > > > > > > > > > because
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalog instance
> already
> > > > exists
> > > > > > in
> > > > > > > > > > catalogs
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > > > > corresponding to
> > > > > > > > > the
> > > > > > > > > > > > > original
> > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it will
> confuse
> > > > users
> > > > > > that
> > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
> > > catalog)`
> > > > > in
> > > > > > the
> > > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This could potentially
> > lead
> > > > to
> > > > > > > > > > confusion. I
> > > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > > > > previously
> > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > > > > add
> > > `registerCatalog(String
> > > > > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)`
> method
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is
> > already
> > > an
> > > > > > > > instance,
> > > > > > > > > > > > adding
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > > > method may not
> > necessarily
> > > > > > result in
> > > > > > > > > lazy
> > > > > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think
> > about
> > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > > > > necessary,
> > > > > > but
> > > > > > > > > > > perhaps
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > > > > should be implemented
> in
> > > the
> > > > > > > > > CatalogStore
> > > > > > > > > > > > when
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance,
> the
> > > > > > > > > FileCatalogStore
> > > > > > > > > > > > could
> > > > > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
> > > while
> > > > > the
> > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need
> the
> > > > > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary
> > to
> > > > > have a
> > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, which
> > saves
> > > > > > Catalog
> > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want
> to
> > > > > > implement
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > > > > additional
> > > > > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > > > > registration
> > > > > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > > > > concern.
> > > > > > > > When
> > > > > > > > > > > > > executing
> > > > > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > > > > statements that
> > reference a
> > > > > > specific
> > > > > > > > > > > Catalog,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > > > > will be initialized if
> it
> > > has
> > > > > not
> > > > > > > > been
> > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > initialization process
> > > fails,
> > > > > > these
> > > > > > > > > > > > statements
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> > > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all
> > have
> > > > the
> > > > > > same
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > name,
> > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> > > instances.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This issue can be
> > resolved
> > > by
> > > > > > > > referring
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > > key principle is that
> the
> > > > > Catalog
> > > > > > > > that
> > > > > > > > > > has
> > > > > > > > > > > > been
> > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > > > > will be given priority
> > for
> > > > > > subsequent
> > > > > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> > > > 10:58 PM
> > > > > > > > Xianxun
> > > > > > > > > > Ye
> > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving
> this
> > > > Flip,
> > > > > > I do
> > > > > > > > > > > believe
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I
> > work
> > > > for
> > > > > > also
> > > > > > > > > > > manages a
> > > > > > > > > > > > > lot
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks
> > becomes
> > > > > slow
> > > > > > > > > because
> > > > > > > > > > of
> > > > > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs
> in
> > > the
> > > > > > user's
> > > > > > > > > > Flink
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
> > > catalogs
> > > > > to
> > > > > > > > > improve
> > > > > > > > > > > the
> > > > > > > > > > > > > > speed
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > > > > After reading
> flip-295, I
> > > > have
> > > > > > some
> > > > > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
> > > catalog
> > > > > with
> > > > > > > > > CREATE
> > > > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
> > used
> > > > by
> > > > > > > > default.
> > > > > > > > > > > What
> > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
> > > implement
> > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the
> same
> > > > > > question
> > > > > > > > with
> > > > > > > > > > > > > Shammon’s
> > > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > > catalogs
> > > > > and
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have
> > the
> > > > > same
> > > > > > > > > catalog
> > > > > > > > > > > > name,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> > > instances.
> > > > > For
> > > > > > > > > example:
> > > > > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1
> ->
> > > > > > hiveCatalog
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> > > > define
> > > > > > > > > `catalogs`
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日
> 11:13，Hang
> > > > Ruan
> > > > > <
> > > > > > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Samrat Deb <de...@gmail.com>.
Hi Feng,
Thank you for providing the proposal. I believe this feature will be highly
valuable.
I have a couple of inquiries:
1. According to the documentation [1], the createCatalog method does not
mention any exceptions being thrown. However, I would like to confirm if it
is always true that there will be no failures in all scenarios. Please let
me know if there is any additional information I may have missed.
2. Regarding the registration process using the `table.catalog-store.kind`
configuration, could you please provide an exhaustive list of the supported
kinds?
It would be great to have a comprehensive understanding of the options
available.
Bests,
Samrat
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
> Hi Hang and Jark
>
> Thank you very much for your reply.
>
> @Jark
> > 1. Could you move the CatalogStore registration API to the "Public
> Interface" section?
>
> Indeed, it is more reasonable.
>
> > 2. We should prefix "table." for the CatalogStore configuration.
>
> Sure, the names that you provided are indeed more logical and easier to
> understand.
>
> > 3. About the open/close method in CatalogStoreFactory.
>
> I also agree with the proposed requirement scenario and design. I have
> already made modifications to the interface.
>
> @Hang
>
> > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> the resource.
>
> I have already added the missing method.
>
> > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>
> Sorry for the misspelling, I updated this part.
>
> > 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
> to init or close its resource
>
> This is indeed a scenario that needs to be considered, and I have added
> relevant methods accordingly.
> Additionally, I have included some relevant description information in the
> documentation to help others understand
> why the open/close methods need to be added.
>
>
> Best,
> Feng
>
> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
>
> > Hi Jing,
> >
> > Thank you for the update.
> >
> > 1. Could you move the CatalogStore registration API to the "Public
> > Interface" section?
> > "Proposed Changes" is more like a place to describe the implementation
> > details.
> >
> > 2. We should prefix "table." for the CatalogStore configuration. Besides,
> > the config key
> > name should be hierarchical[1]. Therefore, it may be better to use:
> > "table.catalog-store.kind"
> > "table.catalog-store.file.path"
> >
> > 3. I think Hang's suggestions make sense.
> >
> > Others look good to me.
> >
> > Best,
> > Jark
> >
> > On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com> wrote:
> >
> > > Hi, Feng.
> > >
> > > Thanks for the update.
> > > The current design looks good to me. I have some minor comments.
> > >
> > > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> > the
> > > resource. For example, when storing the information in MySQL, the store
> > > needs to open and close the connections.
> > >
> > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > >
> > > 3. About the usage in the sql gateway.
> > > Considering the usage in sql gateway, the sql gateway may create a
> > > CatalogStore for each session.
> > > If we are using the MySqlCatalogStore, there would be so many
> > connections.
> > > How can we reuse the connection among these sessions?
> > > I think sql gateway need to maintain a connection pool in
> > > the CatalogStoreFactory and each session get its own connection from
> the
> > > pool when it is created.
> > > Then the `CatalogStoreFactory` may need the `open`/`close` methods to
> > init
> > > or close its resource.
> > > Or is there a better way?
> > >
> > > Best,
> > > Hang
> > >
> > > Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> > >
> > > > Thanks Jingsong.
> > > >
> > > > > Just naming, maybe `createCatalog` in TableEnv
> > > >
> > > > +1 For this, I have already updated FLIP.
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > >
> > > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Feng,
> > > > >
> > > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > > > functions are converted to createxxx from registerxxx.
> > > > >
> > > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi jark, thanks for your suggestion.
> > > > > >
> > > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > > > CatalogStore should be immutable once TableEnv is created.
> > Otherwise,
> > > > > there
> > > > > > might be some data inconsistencies when CatalogStore is changed.
> > > > > >
> > > > > > Yes, We should initialize the CatalogStore when creating the
> > > TableEnv.
> > > > > > Therefore, my current proposal is to add a method to configure
> the
> > > > > > CatalogStore in EnvironmentSettings.
> > > > > >
> > > > > > // Initialize a catalog Store
> > > > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > > > >
> > > > > > // set up the Table API
> > > > > > final EnvironmentSettings settings =
> > > > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > > > .withCatalogStore(catalogStore)
> > > > > > .build();
> > > > > >
> > > > > > final TableEnvironment tableEnv =
> > > > > TableEnvironment.create(settings);
> > > > > >
> > > > > >
> > > > > > > 2. Why does the CatalogStoreFactory interface only have a
> default
> > > > > method,
> > > > > > not an interface method?
> > > > > >
> > > > > > Sorry, While I did refer to the Catalog interface, I agree that
> as
> > a
> > > > new
> > > > > > interface, the CatalogStoreFactory should not have a default
> method
> > > but
> > > > > an
> > > > > > interface method. I have already modified the interface.
> > > > > >
> > > > > > > 3. Please mention the alternative API in Javadoc for the
> > > > > > deprecated`registerCatalog`.
> > > > > > > 4. In the "Compatibility" section, would be better to mention
> the
> > > > > changed
> > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> other
> > > > > > persisted catalog store) is used.
> > > > > >
> > > > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > > > >
> > > > > >
> > > > > > [1].
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > > > The current design looks pretty good to me. I just have some
> > minor
> > > > > > > comments.
> > > > > > >
> > > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > > CatalogStore
> > > > > > > should be immutable once TableEnv is created. Otherwise, there
> > > might
> > > > be
> > > > > > > some data inconsistencies when CatalogStore is changed.
> > > > > > >
> > > > > > > 2. Why does the CatalogStoreFactory interface only have a
> default
> > > > > method,
> > > > > > > not an interface method?
> > > > > > >
> > > > > > > 3. Please mention the alternative API in Javadoc for the
> > deprecated
> > > > > > > `registerCatalog`.
> > > > > > >
> > > > > > > 4. In the "Compatibility" section, would be better to mention
> the
> > > > > changed
> > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> > other
> > > > > > > persisted catalog store) is used.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Jark
> > > > > > >
> > > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi ， thanks all for reviewing the flip.
> > > > > > > >
> > > > > > > > @Ron
> > > > > > > >
> > > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore
> method,
> > > do
> > > > we
> > > > > > > need
> > > > > > > > to provide a default implementation?
> > > > > > > >
> > > > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> > > > create
> > > > > an
> > > > > > > > InMemoryCatalogStore.
> > > > > > > >
> > > > > > > > > If we get a Catalog from CatalogStore, after initializing
> > it,
> > > > > whether
> > > > > > > we
> > > > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > > > >
> > > > > > > > Yes, in the current design, catalogs are stored as snapshots,
> > and
> > > > > once
> > > > > > > > initialized, the Catalog will be placed in the Map<String,
> > > Catalog>
> > > > > > > > catalogs.
> > > > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> > > primary
> > > > > > > source
> > > > > > > > for obtaining the corresponding Catalog.
> > > > > > > >
> > > > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > > > `catalog.store.path`?
> > > > > > > >
> > > > > > > > I think it is okay. Adding "sql" at the beginning may seem a
> > bit
> > > > > > > strange. I
> > > > > > > > will update the FLIP.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > @Shammon
> > > > > > > >
> > > > > > > > Thank you for the review. I have made the necessary
> > corrections.
> > > > > > > > Regarding the modifications made to the Public Interface, I
> > have
> > > > also
> > > > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> zjureel@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi feng,
> > > > > > > > >
> > > > > > > > > Thanks for updating, I have some minor comments
> > > > > > > > >
> > > > > > > > > 1. The modification of `CatalogManager` should not be in
> > > `Public
> > > > > > > > > Interfaces`, it is not a public interface.
> > > > > > > > >
> > > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > > > > `CatalogStoreFactory`
> > > > > > > > >
> > > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
> > should
> > > be
> > > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > > > catalogStore.get(catalogName);`
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shammon FY
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <
> ron9.liu@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Feng
> > > > > > > > > >
> > > > > > > > > > Thanks for driving this FLIP, this proposal is very
> useful
> > > for
> > > > > > > catalog
> > > > > > > > > > management.
> > > > > > > > > > I have some small questions:
> > > > > > > > > >
> > > > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> > > method,
> > > > > do we
> > > > > > > > > need
> > > > > > > > > > to provide a default implementation?
> > > > > > > > > > 2. If we get Catalog from CatalogStore, after
> initializing
> > > it,
> > > > > > > whether
> > > > > > > > we
> > > > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Ron
> > > > > > > > > >
> > > > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > > > > >
> > > > > > > > > > > Hi yuxia
> > > > > > > > > > >
> > > > > > > > > > > > But from the code in Proposed Changes, once we
> > register
> > > > the
> > > > > > > > Catalog,
> > > > > > > > > > we
> > > > > > > > > > > initialize it and open it. right?
> > > > > > > > > > >
> > > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> > > original
> > > > > > > CREATE
> > > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > > > registerCatalog
> > > > > > > > so
> > > > > > > > > > > that parameter validation can be performed.
> > > > > > > > > > >
> > > > > > > > > > > In the current design, lazy initialization is mainly
> > > > reflected
> > > > > in
> > > > > > > > > > > getCatalog. If CatalogStore has already saved some
> > catalog
> > > > > > > > > > configurations,
> > > > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > > > luoyuxia@alumni.sjtu.edu.cn
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi, Feng.
> > > > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > > > initialization*.
> > > > > > > > If
> > > > > > > > > > i'm
> > > > > > > > > > > > wrong, please correct me.
> > > > > > > > > > > >
> > > > > > > > > > > > IIUC, lazy initialization means only you need to
> access
> > > the
> > > > > > > > catalog,
> > > > > > > > > > then
> > > > > > > > > > > > you initialize it. But from the code in Proposed
> > Changes,
> > > > > once we
> > > > > > > > > > > register
> > > > > > > > > > > > the Catalog,
> > > > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Yuxia
> > > > > > > > > > > >
> > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > > Catalog
> > > > > > > > > > configuration
> > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > > > >
> > > > > > > > > > > > One of the major changes is that current design will
> > > > provide
> > > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead
> of a
> > > > > cache,
> > > > > > > > which
> > > > > > > > > > > means
> > > > > > > > > > > > once it has been initialized, any changes done by
> other
> > > > > sessions
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > affect it. Point 6 described follow-up options for
> > > further
> > > > > > > > > improvement.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > > > jinfeng1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all, I would like to update you on the latest
> > > progress
> > > > > of
> > > > > > > the
> > > > > > > > > > FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon
> FY,
> > > > > ShengKai
> > > > > > > > Fang
> > > > > > > > > > > and I
> > > > > > > > > > > > > had an offline discussion regarding the overall
> > > solution
> > > > > for
> > > > > > > > Flink
> > > > > > > > > > > > > CatalogStore. We have reached a consensus and I
> have
> > > > > updated
> > > > > > > the
> > > > > > > > > > final
> > > > > > > > > > > > > solution in FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > > > configuration
> > > > > > > > > similar
> > > > > > > > > > > to
> > > > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > > > 2.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > > > storeCatalog(String
> > > > > > > > > > > > > catalogName, CatalogDescriptor) and
> > > CatalogDescriptor
> > > > > > > > > > > > getCatalog(String)
> > > > > > > > > > > > > will both use CatalogDescriptor instead of
> Catalog
> > > > > instance.
> > > > > > > > > This
> > > > > > > > > > > way,
> > > > > > > > > > > > > CatalogStore will only be responsible for saving
> > and
> > > > > > > > retrieving
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > configurations without having to initialize
> > > catalogs.
> > > > > > > > > > > > > 3.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The default registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > > > function in CatalogManager will be marked as
> > > > deprecated.
> > > > > > > > > > > > > 4.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A new function registerCatalog(String
> catalogName,
> > > > > > > > > > CatalogDescriptor
> > > > > > > > > > > > > catalog) will be added to serve as the default
> > > > > registration
> > > > > > > > > > function
> > > > > > > > > > > > for
> > > > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > > > 5.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager
> will
> > > > > remain
> > > > > > > > > > unchanged
> > > > > > > > > > > > and
> > > > > > > > > > > > > save initialized catalogs.This means that
> deletion
> > > > > > > operations
> > > > > > > > > from
> > > > > > > > > > > one
> > > > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > > > 6.
> > > > > > > > > > > > >
> > > > > > > > > > > > > To support multi-session synchronization
> scenarios
> > > for
> > > > > > > > deletions
> > > > > > > > > > > later
> > > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > > > configurable.There
> > > > > > > > > > may
> > > > > > > > > > > > be
> > > > > > > > > > > > > three possible situations:
> > > > > > > > > > > > >
> > > > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > > > >
> > > > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > > > automatically
> > > > > > > > clear
> > > > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > c.No caching of any instances; each call to
> > > getCatalog
> > > > > > > > creates a
> > > > > > > > > > new
> > > > > > > > > > > > > instance.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you very much for your attention and
> > suggestions
> > > on
> > > > > this
> > > > > > > > > > FLIP. A
> > > > > > > > > > > > > special thanks to Hang Ruan for his suggestions on
> > the
> > > > > entire
> > > > > > > > > design
> > > > > > > > > > > and
> > > > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you have any further suggestions or feedback
> about
> > > > this
> > > > > FLIP
> > > > > > > > > > please
> > > > > > > > > > > > feel
> > > > > > > > > > > > > free to share.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for improving the FLIP. It looks good to
> me.
> > > We
> > > > > could
> > > > > > > > > still
> > > > > > > > > > > > > > reconsider in the future how to provide more
> common
> > > > > built-in
> > > > > > > > > cache
> > > > > > > > > > > > > > functionality in CatalogManager so that not every
> > > > > > > CatalogSotre
> > > > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > > > jinfeng1995@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There might be more such issues. I would
> > suggest
> > > > you
> > > > > > > > > > completely
> > > > > > > > > > > > walk
> > > > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I am very sorry for my carelessness and at the
> > same
> > > > > time, I
> > > > > > > > > > greatly
> > > > > > > > > > > > > > > appreciate your careful review. I have
> thoroughly
> > > > > checked
> > > > > > > the
> > > > > > > > > > > entire
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> > > > design,
> > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > could work without Optional CatalogStore being
> > > > > configured.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, in the original design, CatalogStore was
> not
> > > > > necessary
> > > > > > > > > > because
> > > > > > > > > > > > > > > CatalogManager used Map<String, Catalog>
> catalogs
> > > to
> > > > > store
> > > > > > > > > > catalog
> > > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > However, this caused inconsistency issues.
> > > > Therefore, I
> > > > > > > > > modified
> > > > > > > > > > > this
> > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > of the design and removed Map<String, Catalog>
> > > > catalogs
> > > > > > > from
> > > > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > > > At the same time, InMemoryCatalog will serve
> as
> > > the
> > > > > > > default
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > to save catalogs in memory and replace the
> > > > > functionality of
> > > > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > > > The previous plan that kept
> > > > > Map<String,Catalog>catalogs has
> > > > > > > > > been
> > > > > > > > > > > > moved
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are still many places contain
> > inconsistent
> > > > > content,
> > > > > > > > > e.g.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > > > > 2. The java comment of the method
> > > > > registerCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not
> > tell
> > > > > what the
> > > > > > > > > > method
> > > > > > > > > > > is
> > > > > > > > > > > > > > > doing.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There might be more such issues. I would
> > suggest
> > > > you
> > > > > > > > > completely
> > > > > > > > > > > > walk
> > > > > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean
> > that
> > > > you
> > > > > will
> > > > > > > > > build
> > > > > > > > > > > the
> > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > functionality in the CatalogStore? This is a
> > very
> > > > > > > different
> > > > > > > > > > > design
> > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > from what the current FLIP described. If I am
> > not
> > > > > > > mistaken,
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > current FLIP design, CatalogManager could
> work
> > > > > without
> > > > > > > > > Optional
> > > > > > > > > > > > > > > > CatalogStore being configured. That is the
> > reason
> > > > > why I
> > > > > > > > > > mentioned
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > last email that the example code wrt the
> > Optional
> > > > > > > > > CatalogStore
> > > > > > > > > > is
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There are still some NIT issues in the
> FLIP
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you very much for the careful
> review. I
> > > > have
> > > > > > > > already
> > > > > > > > > > made
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > > multi-instance
> > > > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > > > > scenarios
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > design. I am thinking whether we should
> > remove
> > > > > > > > 'Map<String,
> > > > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > > > InMemoryCatalogStore.
> > > > > > > > > This
> > > > > > > > > > > > way,
> > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > can implement their own CatalogStore to
> > achieve
> > > > > > > > > > multi-instance
> > > > > > > > > > > > > > > > > non-conflicting scenarios. What do you
> think?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There are
> > > still
> > > > > some
> > > > > > > > NIT
> > > > > > > > > > > issues
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore
> has
> > > been
> > > > > used
> > > > > > > as
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > instead of Optional in the code example.
> It
> > > > > should be
> > > > > > > > > fine
> > > > > > > > > > to
> > > > > > > > > > > > use
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > pseudo code for now and update it after
> you
> > > > > submit
> > > > > > > the
> > > > > > > > > PR.
> > > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used
> somewhere
> > in
> > > > the
> > > > > > > > > rejected
> > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > > > > consistent.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > multi-instance
> > > > > > > > > > > > scenarios,
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > sure if this is the intended behaviour.
> If
> > > > > > > Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > used as a cache, it should be invalid,
> once
> > > the
> > > > > > > related
> > > > > > > > > > > catalog
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > removed from the CatalogStore by another
> > > > > instance.
> > > > > > > Did
> > > > > > > > I
> > > > > > > > > > miss
> > > > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin
> <
> > > > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > > > I think this is a good function name, I
> > > have
> > > > > > > updated
> > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Some common cache features should
> be
> > > > > > > implemented
> > > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> > > alternative
> > > > 1
> > > > > > > turns
> > > > > > > > > out
> > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > appropriate later, I will improve this
> > part
> > > > of
> > > > > the
> > > > > > > > > > content.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > As the above discussion moves
> forward,
> > > the
> > > > > > > option 2
> > > > > > > > > > > > solution
> > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
> > offline,
> > > > we
> > > > > > > think
> > > > > > > > > that
> > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > > > inconsistency
> > > > > > > > > issues.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There are some inconsistent
> > descriptions
> > > in
> > > > > the
> > > > > > > > > > content.
> > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > > > I will do my best to improve the
> document
> > > and
> > > > > > > > > appreciate
> > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > can you put the unselected option in
> > > > > `Rejected
> > > > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > > > > Alternatives`.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon
> > FY
> > > <
> > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I found there are two options in
> > > `Proposed
> > > > > > > > Changes`,
> > > > > > > > > > can
> > > > > > > > > > > > you
> > > > > > > > > > > > > > put
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > > > Alternatives`? I
> > > > > > > > think
> > > > > > > > > > > this
> > > > > > > > > > > > > may
> > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing
> Ge
> > > > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am
> > > still
> > > > > > > confused
> > > > > > > > > > after
> > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > the thread with following
> questions:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
> > registerCatalog()
> > > > and
> > > > > > > > > > > addCatalog()
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > > > difference based on their names.
> One
> > of
> > > > > them is
> > > > > > > > > > > > responsible
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > persistence. How about
> > > persistCatalog()?
> > > > > > > > > > > > > > > > > > > > > 2. As you mentioned that
> Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs
> > > > > > > > > > > is
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > > > persistence.
> > > > > > > I
> > > > > > > > > > would
> > > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > > > their purpose conceptually and
> > clearly
> > > in
> > > > > the
> > > > > > > > FLIP.
> > > > > > > > > > > Some
> > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > features should be implemented,
> i.e.
> > > data
> > > > > in
> > > > > > > the
> > > > > > > > > > cache
> > > > > > > > > > > > and
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog
> instance
> > > > > should be
> > > > > > > > > found
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the cache(either it has been
> > > initialized
> > > > > or it
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > > > consistency
> > > > > will
> > > > > > > > be
> > > > > > > > > > > taken
> > > > > > > > > > > > > care
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> > > forward,
> > > > > the
> > > > > > > > > option
> > > > > > > > > > 2
> > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> > > because,
> > > > > afaiu,
> > > > > > > > > > issues
> > > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > previously with option 1 are not
> > solved
> > > > > yet. Do
> > > > > > > > you
> > > > > > > > > > > still
> > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > > > both options and ask for
> suggestions
> > > for
> > > > > both
> > > > > > > of
> > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP,
> there
> > > are
> > > > > some
> > > > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the content. Would you like to
> clean
> > > > them
> > > > > up?
> > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng
> > > Jin <
> > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thank you for your response, and
> I
> > > > > completely
> > > > > > > > > agree
> > > > > > > > > > > > with
> > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > > > Initially, I may have over
> > > complicated
> > > > > the
> > > > > > > > whole
> > > > > > > > > > > issue.
> > > > > > > > > > > > > > First
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > > > we need to consider the
> persistence
> > > of
> > > > > the
> > > > > > > > > > Catalog's
> > > > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > > > If we only need to provide
> > > persistence
> > > > > for
> > > > > > > > > Catalog
> > > > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > add a toConfiguration method to
> the
> > > > > Catalog
> > > > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > > > > instance
> > > > > > > to a
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > > properties, and the default
> > > > > implementation
> > > > > > > will
> > > > > > > > > > throw
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > > > > properties
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > > > * @return a map containing the
> > > > > properties
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > > > * @throws
> > > > > UnsupportedOperationException if
> > > > > > > > the
> > > > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > > > * the default implementation
> of
> > > this
> > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > > > toProperties()
> > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The specific process is as
> follows:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > > > > CatalogStore,
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > will be called when registering a
> > > > Catalog
> > > > > > > > > instance
> > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > > catalog).
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > will be converted to a
> Map<String,
> > > > > String>
> > > > > > > > > > properties
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> > > instances
> > > > > do
> > > > > > > not
> > > > > > > > > > > > implement
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > > > CatalogStore,
> > > > > > > > the
> > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > > > consistency
> > > > > with
> > > > > > > > the
> > > > > > > > > > > > > original
> > > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > > the same time can also avoid
> > > conflicts
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
> > itself
> > > > and
> > > > > > > > provide
> > > > > > > > > a
> > > > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > implementation for lazy loading,
> > such
> > > > as
> > > > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > > > > implements
> > > > > > > > > > Catalog
> > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> > > > String>
> > > > > > > > > > properties)
> > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I have added this design to the
> > FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> > > Shammon
> > > > > FY <
> > > > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > > > functionality
> > > > > > > > of
> > > > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > different operations of
> multiple
> > > > > > > > > > `registerCatalog`
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> > > implementation
> > > > in
> > > > > > > Trino
> > > > > > > > > is
> > > > > > > > > > > also
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only
> > has
> > > > one
> > > > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > catalog to memory and
> > > `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> > > another
> > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> > > > Catalog>
> > > > > > > > > catalogs`
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you
> mentioned
> > > > > before,
> > > > > > > the
> > > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > > > `CatalogManager`.
> > > > > > > How
> > > > > > > > > > about
> > > > > > > > > > > > > > saving
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs`
> > and
> > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > > > when it is registered or added
> in
> > > > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM
> > Feng
> > > > > Jin <
> > > > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I
> am
> > > very
> > > > > sorry
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
> > original
> > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > > pre-discussion
> > > > > [1]
> > > > > > > for
> > > > > > > > > > this
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > > > > discussion
> > > > > > > > > before.
> > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > whole solution, I found that
> > the
> > > > > logic of
> > > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > > need to change much. *We
> cannot
> > > > only
> > > > > > > > persist
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > > > registerCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > > > not be enough to save
> Catalogs,
> > > > > because
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > end
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > > the configurations
> > corresponding
> > > to
> > > > > the
> > > > > > > > > Catalog
> > > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > > > CatalogStore
> > > > > > > > > interface
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> > > part, I
> > > > > also
> > > > > > > > took
> > > > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> handling
> > > > > catalog
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > > > Thank you for the
> suggestion. I
> > > > will
> > > > > > > > provide
> > > > > > > > > a
> > > > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > code examples for this part,
> > and
> > > > add
> > > > > it
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > > `addCatalog`
> > > > > > > > still
> > > > > > > > > > > looks
> > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > catalog
> > > > to
> > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > > > personal
> > > > > > > > > opinion,
> > > > > > > > > > > you
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
> > intuitive
> > > > to
> > > > > me.
> > > > > > > I
> > > > > > > > > > don't
> > > > > > > > > > > > see
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
> > statement,
> > > > > which
> > > > > > > > > method
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > `addCatalog`?
> > > > > I'm
> > > > > > > > > > wondering
> > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my
> original
> > > > > design
> > > > > > > was
> > > > > > > > to
> > > > > > > > > > add
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> > > > disrupt
> > > > > the
> > > > > > > > > > existing
> > > > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > > > configuration to
> > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the
> > same
> > > > time
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary
> to
> > > > > provide a
> > > > > > > > > > default
> > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > interface
> > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> > > otherwise
> > > > we
> > > > > > > would
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > > > Map to store the
> configuration
> > > for
> > > > > lazy
> > > > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > registration
> > > > > for
> > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > I don't think
> > > > registerCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> > > interface
> > > > > > > because
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > initialization
> > > > > > > > for
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > > > inaccurate. It
> > > > > > > > > > should
> > > > > > > > > > > be
> > > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > > > registration. I
> > > > > > > > have
> > > > > > > > > > > > already
> > > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> 4:27 PM
> > > > > Shammon
> > > > > > > FY <
> > > > > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > > > `registerCatalog`
> > > > > > > > > > method
> > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
> > method
> > > > > named
> > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> > > memory
> > > > > > > catalog
> > > > > > > > > is a
> > > > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> > > exposed
> > > > to
> > > > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > > pre-discussion
> > > > > [1]
> > > > > > > for
> > > > > > > > > > this
> > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
> > conclusion
> > > > of
> > > > > that
> > > > > > > > > > > > discussion
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> > > interface
> > > > > and
> > > > > > > > > > implement
> > > > > > > > > > > it
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > > as memory, file and
> external
> > > > > system.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
> > between
> > > > the
> > > > > > > > current
> > > > > > > > > > FLIP
> > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > > > What about the proposal of
> > the
> > > > > > > discussion
> > > > > > > > > in
> > > > > > > > > > > > thread
> > > > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > >
> > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> > 3:41 PM
> > > > > yuxia <
> > > > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving
> > this
> > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> > handling
> > > > > catalog
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
> > it'll
> > > be
> > > > > > > better
> > > > > > > > to
> > > > > > > > > > > > explain
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > these methods and give a
> > > brief
> > > > > > > example
> > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > > `addCatalog`
> > > > > > > > > still
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > > this method is for
> storing
> > > > > catalog to
> > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> > very
> > > > > personal
> > > > > > > > > > > opinion,
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > > > statement,
> > > > > > > which
> > > > > > > > > > method
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > > `addCatalog`?
> > > > > > > I'm
> > > > > > > > > > > > wondering
> > > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really
> neccessary
> > to
> > > > > > > provide a
> > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > > registration
> > > > > > > for
> > > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog
> > > with
> > > > > CREATE
> > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
> used
> > > by
> > > > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
> > registration
> > > is
> > > > > the
> > > > > > > > > default
> > > > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > > > synchronous
> > > > > > > > > > > > registration
> > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > > > Will both method
> > `addCatalog`
> > > > and
> > > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > > > registration,
> > > > > > > it
> > > > > > > > > may
> > > > > > > > > > > well
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > > > executes successfully,
> but
> > > then
> > > > > the
> > > > > > > > > > following
> > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail
> > to
> > > > > open.
> > > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break
> change
> > > > which
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying
> asynchronous
> > > > > > > > > registration, I
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to
> open
> > > or
> > > > > > > register
> > > > > > > > > > > catalog
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance
> will
> > be
> > > > > > > > initialized
> > > > > > > > > > if
> > > > > > > > > > > it
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > > > yet. If the
> initialization
> > > > > process
> > > > > > > > fails,
> > > > > > > > > > > these
> > > > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > > initialization
> > > > > > > > > for
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > > > jinfeng1995@gmail.com
> > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > > > dev@flink.apache.org
> > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日
> 下午
> > > > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP
> 295:
> > > > > Support
> > > > > > > > > > > persistence
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
> > registration
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you
> all
> > > for
> > > > > your
> > > > > > > > > > interest
> > > > > > > > > > > in
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog
> > > with
> > > > > the
> > > > > > > same
> > > > > > > > > > name
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a
> crucial
> > > > > point.
> > > > > > > > Based
> > > > > > > > > on
> > > > > > > > > > > my
> > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> > > Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs
> > and
> > > > > have
> > > > > > > the
> > > > > > > > > > > highest
> > > > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > There are three methods
> > that
> > > > can
> > > > > > > affect
> > > > > > > > > the
> > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
> > catalog
> > > > > instance
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2.
> unregisterCatalog(String
> > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > catalog
> > > > > > > > instance
> > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
> > Catalog>
> > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This method first
> retrieves
> > > the
> > > > > > > > > > corresponding
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > > > > catalogs. If
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > retrieves the
> corresponding
> > > > > > > > configuration
> > > > > > > > > > > from
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts
> > the
> > > > > > > > initialized
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > The following two methods
> > > only
> > > > > modify
> > > > > > > > the
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > > > catalogName,
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > > > properties
> > > > > to
> > > > > > > the
> > > > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> > > catalogName
> > > > > with
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > > specified
> > > > > > > > > > > configuration
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > > > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > The following are
> possible
> > > > > conflict
> > > > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > > > > catalogName
> > > > > > > > > > already
> > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in
> the
> > > > > > > Map<String,
> > > > > > > > > > > > Catalog>,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly
> > saved
> > > > to
> > > > > the
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > > > > catalogName
> > > > > > > > > > already
> > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > > > Map<String,
> > > > > > > > > Catalog>,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
> > catalog)
> > > > > method
> > > > > > > > will
> > > > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > > > > catalogName
> > > > > > > > > > already
> > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > > String> properties)
> method
> > > can
> > > > > > > directly
> > > > > > > > > > save
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > > > getCatalog(String
> > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > > use the new properties
> for
> > > > > > > > initialization
> > > > > > > > > > > > because
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > > catalog instance already
> > > exists
> > > > > in
> > > > > > > > > catalogs
> > > > > > > > > > > and
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > > > corresponding to
> > > > > > > > the
> > > > > > > > > > > > original
> > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse
> > > users
> > > > > that
> > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
> > catalog)`
> > > > in
> > > > > the
> > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > catalogName,
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This could potentially
> lead
> > > to
> > > > > > > > > confusion. I
> > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > > > previously
> > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > > > add
> > `registerCatalog(String
> > > > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is
> already
> > an
> > > > > > > instance,
> > > > > > > > > > > adding
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > > > registerCatalog(String
> > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > > method may not
> necessarily
> > > > > result in
> > > > > > > > lazy
> > > > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think
> about
> > > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > > > necessary,
> > > > > but
> > > > > > > > > > perhaps
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > > > should be implemented in
> > the
> > > > > > > > CatalogStore
> > > > > > > > > > > when
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > > > > FileCatalogStore
> > > > > > > > > > > could
> > > > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
> > while
> > > > the
> > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary
> to
> > > > have a
> > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > > > implementation, which
> saves
> > > > > Catalog
> > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > > > > implement
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > > > additional
> > > > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > > > registration
> > > > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > > > concern.
> > > > > > > When
> > > > > > > > > > > > executing
> > > > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > > > statements that
> reference a
> > > > > specific
> > > > > > > > > > Catalog,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > > > will be initialized if it
> > has
> > > > not
> > > > > > > been
> > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > initialization process
> > fails,
> > > > > these
> > > > > > > > > > > statements
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> > > Catalog>
> > > > > > > catalogs
> > > > > > > > > and
> > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all
> have
> > > the
> > > > > same
> > > > > > > > > > catalog
> > > > > > > > > > > > > name,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> > instances.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This issue can be
> resolved
> > by
> > > > > > > referring
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> > > > Catalog
> > > > > > > that
> > > > > > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > > > will be given priority
> for
> > > > > subsequent
> > > > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> > > 10:58 PM
> > > > > > > Xianxun
> > > > > > > > > Ye
> > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this
> > > Flip,
> > > > > I do
> > > > > > > > > > believe
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I
> work
> > > for
> > > > > also
> > > > > > > > > > manages a
> > > > > > > > > > > > lot
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks
> becomes
> > > > slow
> > > > > > > > because
> > > > > > > > > of
> > > > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in
> > the
> > > > > user's
> > > > > > > > > Flink
> > > > > > > > > > > SQL
> > > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
> > catalogs
> > > > to
> > > > > > > > improve
> > > > > > > > > > the
> > > > > > > > > > > > > speed
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I
> > > have
> > > > > some
> > > > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
> > catalog
> > > > with
> > > > > > > > CREATE
> > > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
> used
> > > by
> > > > > > > default.
> > > > > > > > > > What
> > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
> > implement
> > > > > > > > asynchronous
> > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > > > > question
> > > > > > > with
> > > > > > > > > > > > Shammon’s
> > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > catalogs
> > > > and
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have
> the
> > > > same
> > > > > > > > catalog
> > > > > > > > > > > name,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> > instances.
> > > > For
> > > > > > > > example:
> > > > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > > > > hiveCatalog
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> > > define
> > > > > > > > `catalogs`
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang
> > > Ruan
> > > > <
> > > > > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Hang and Jark
Thank you very much for your reply.
@Jark
> 1. Could you move the CatalogStore registration API to the "Public
Interface" section?
Indeed, it is more reasonable.
> 2. We should prefix "table." for the CatalogStore configuration.
Sure, the names that you provided are indeed more logical and easier to
understand.
> 3. About the open/close method in CatalogStoreFactory.
I also agree with the proposed requirement scenario and design. I have
already made modifications to the interface.
@Hang
> 1. The `CatalogStore` need the `open`/`close` methods to init or close
the resource.
I have already added the missing method.
> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
Sorry for the misspelling, I updated this part.
> 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
to init or close its resource
This is indeed a scenario that needs to be considered, and I have added
relevant methods accordingly.
Additionally, I have included some relevant description information in the
documentation to help others understand
why the open/close methods need to be added.
Best,
Feng
On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> Hi Jing,
>
> Thank you for the update.
>
> 1. Could you move the CatalogStore registration API to the "Public
> Interface" section?
> "Proposed Changes" is more like a place to describe the implementation
> details.
>
> 2. We should prefix "table." for the CatalogStore configuration. Besides,
> the config key
> name should be hierarchical[1]. Therefore, it may be better to use:
> "table.catalog-store.kind"
> "table.catalog-store.file.path"
>
> 3. I think Hang's suggestions make sense.
>
> Others look good to me.
>
> Best,
> Jark
>
> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com> wrote:
>
> > Hi, Feng.
> >
> > Thanks for the update.
> > The current design looks good to me. I have some minor comments.
> >
> > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> the
> > resource. For example, when storing the information in MySQL, the store
> > needs to open and close the connections.
> >
> > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >
> > 3. About the usage in the sql gateway.
> > Considering the usage in sql gateway, the sql gateway may create a
> > CatalogStore for each session.
> > If we are using the MySqlCatalogStore, there would be so many
> connections.
> > How can we reuse the connection among these sessions?
> > I think sql gateway need to maintain a connection pool in
> > the CatalogStoreFactory and each session get its own connection from the
> > pool when it is created.
> > Then the `CatalogStoreFactory` may need the `open`/`close` methods to
> init
> > or close its resource.
> > Or is there a better way?
> >
> > Best,
> > Hang
> >
> > Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> >
> > > Thanks Jingsong.
> > >
> > > > Just naming, maybe `createCatalog` in TableEnv
> > >
> > > +1 For this, I have already updated FLIP.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Feng,
> > > >
> > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > > functions are converted to createxxx from registerxxx.
> > > >
> > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi jark, thanks for your suggestion.
> > > > >
> > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > > CatalogStore should be immutable once TableEnv is created.
> Otherwise,
> > > > there
> > > > > might be some data inconsistencies when CatalogStore is changed.
> > > > >
> > > > > Yes, We should initialize the CatalogStore when creating the
> > TableEnv.
> > > > > Therefore, my current proposal is to add a method to configure the
> > > > > CatalogStore in EnvironmentSettings.
> > > > >
> > > > > // Initialize a catalog Store
> > > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > > >
> > > > > // set up the Table API
> > > > > final EnvironmentSettings settings =
> > > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > > .withCatalogStore(catalogStore)
> > > > > .build();
> > > > >
> > > > > final TableEnvironment tableEnv =
> > > > TableEnvironment.create(settings);
> > > > >
> > > > >
> > > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > > method,
> > > > > not an interface method?
> > > > >
> > > > > Sorry, While I did refer to the Catalog interface, I agree that as
> a
> > > new
> > > > > interface, the CatalogStoreFactory should not have a default method
> > but
> > > > an
> > > > > interface method. I have already modified the interface.
> > > > >
> > > > > > 3. Please mention the alternative API in Javadoc for the
> > > > > deprecated`registerCatalog`.
> > > > > > 4. In the "Compatibility" section, would be better to mention the
> > > > changed
> > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > > persisted catalog store) is used.
> > > > >
> > > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > > >
> > > > >
> > > > > [1].
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > > The current design looks pretty good to me. I just have some
> minor
> > > > > > comments.
> > > > > >
> > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > CatalogStore
> > > > > > should be immutable once TableEnv is created. Otherwise, there
> > might
> > > be
> > > > > > some data inconsistencies when CatalogStore is changed.
> > > > > >
> > > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > > method,
> > > > > > not an interface method?
> > > > > >
> > > > > > 3. Please mention the alternative API in Javadoc for the
> deprecated
> > > > > > `registerCatalog`.
> > > > > >
> > > > > > 4. In the "Compatibility" section, would be better to mention the
> > > > changed
> > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> other
> > > > > > persisted catalog store) is used.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > >
> > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi ， thanks all for reviewing the flip.
> > > > > > >
> > > > > > > @Ron
> > > > > > >
> > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore method,
> > do
> > > we
> > > > > > need
> > > > > > > to provide a default implementation?
> > > > > > >
> > > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> > > create
> > > > an
> > > > > > > InMemoryCatalogStore.
> > > > > > >
> > > > > > > > If we get a Catalog from CatalogStore, after initializing
> it,
> > > > whether
> > > > > > we
> > > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > > >
> > > > > > > Yes, in the current design, catalogs are stored as snapshots,
> and
> > > > once
> > > > > > > initialized, the Catalog will be placed in the Map<String,
> > Catalog>
> > > > > > > catalogs.
> > > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> > primary
> > > > > > source
> > > > > > > for obtaining the corresponding Catalog.
> > > > > > >
> > > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > > `catalog.store.path`?
> > > > > > >
> > > > > > > I think it is okay. Adding "sql" at the beginning may seem a
> bit
> > > > > > strange. I
> > > > > > > will update the FLIP.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > @Shammon
> > > > > > >
> > > > > > > Thank you for the review. I have made the necessary
> corrections.
> > > > > > > Regarding the modifications made to the Public Interface, I
> have
> > > also
> > > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi feng,
> > > > > > > >
> > > > > > > > Thanks for updating, I have some minor comments
> > > > > > > >
> > > > > > > > 1. The modification of `CatalogManager` should not be in
> > `Public
> > > > > > > > Interfaces`, it is not a public interface.
> > > > > > > >
> > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > > > `CatalogStoreFactory`
> > > > > > > >
> > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
> should
> > be
> > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > > catalogStore.get(catalogName);`
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shammon FY
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Feng
> > > > > > > > >
> > > > > > > > > Thanks for driving this FLIP, this proposal is very useful
> > for
> > > > > > catalog
> > > > > > > > > management.
> > > > > > > > > I have some small questions:
> > > > > > > > >
> > > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> > method,
> > > > do we
> > > > > > > > need
> > > > > > > > > to provide a default implementation?
> > > > > > > > > 2. If we get Catalog from CatalogStore, after initializing
> > it,
> > > > > > whether
> > > > > > > we
> > > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > > > >
> > > > > > > > > > Hi yuxia
> > > > > > > > > >
> > > > > > > > > > > But from the code in Proposed Changes, once we
> register
> > > the
> > > > > > > Catalog,
> > > > > > > > > we
> > > > > > > > > > initialize it and open it. right?
> > > > > > > > > >
> > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> > original
> > > > > > CREATE
> > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > > registerCatalog
> > > > > > > so
> > > > > > > > > > that parameter validation can be performed.
> > > > > > > > > >
> > > > > > > > > > In the current design, lazy initialization is mainly
> > > reflected
> > > > in
> > > > > > > > > > getCatalog. If CatalogStore has already saved some
> catalog
> > > > > > > > > configurations,
> > > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > > luoyuxia@alumni.sjtu.edu.cn
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Feng.
> > > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > > initialization*.
> > > > > > > If
> > > > > > > > > i'm
> > > > > > > > > > > wrong, please correct me.
> > > > > > > > > > >
> > > > > > > > > > > IIUC, lazy initialization means only you need to access
> > the
> > > > > > > catalog,
> > > > > > > > > then
> > > > > > > > > > > you initialize it. But from the code in Proposed
> Changes,
> > > > once we
> > > > > > > > > > register
> > > > > > > > > > > the Catalog,
> > > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Yuxia
> > > > > > > > > > >
> > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > Catalog
> > > > > > > > > configuration
> > > > > > > > > > > and asynchronous registration
> > > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > > >
> > > > > > > > > > > One of the major changes is that current design will
> > > provide
> > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> > > > cache,
> > > > > > > which
> > > > > > > > > > means
> > > > > > > > > > > once it has been initialized, any changes done by other
> > > > sessions
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > affect it. Point 6 described follow-up options for
> > further
> > > > > > > > improvement.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > > jinfeng1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all, I would like to update you on the latest
> > progress
> > > > of
> > > > > > the
> > > > > > > > > FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> > > > ShengKai
> > > > > > > Fang
> > > > > > > > > > and I
> > > > > > > > > > > > had an offline discussion regarding the overall
> > solution
> > > > for
> > > > > > > Flink
> > > > > > > > > > > > CatalogStore. We have reached a consensus and I have
> > > > updated
> > > > > > the
> > > > > > > > > final
> > > > > > > > > > > > solution in FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > >
> > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > > configuration
> > > > > > > > similar
> > > > > > > > > > to
> > > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > > 2.
> > > > > > > > > > > >
> > > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > > storeCatalog(String
> > > > > > > > > > > > catalogName, CatalogDescriptor) and
> > CatalogDescriptor
> > > > > > > > > > > getCatalog(String)
> > > > > > > > > > > > will both use CatalogDescriptor instead of Catalog
> > > > instance.
> > > > > > > > This
> > > > > > > > > > way,
> > > > > > > > > > > > CatalogStore will only be responsible for saving
> and
> > > > > > > retrieving
> > > > > > > > > > > catalog
> > > > > > > > > > > > configurations without having to initialize
> > catalogs.
> > > > > > > > > > > > 3.
> > > > > > > > > > > >
> > > > > > > > > > > > The default registerCatalog(String catalogName,
> > > Catalog
> > > > > > > catalog)
> > > > > > > > > > > > function in CatalogManager will be marked as
> > > deprecated.
> > > > > > > > > > > > 4.
> > > > > > > > > > > >
> > > > > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > > > > CatalogDescriptor
> > > > > > > > > > > > catalog) will be added to serve as the default
> > > > registration
> > > > > > > > > function
> > > > > > > > > > > for
> > > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > > 5.
> > > > > > > > > > > >
> > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> > > > remain
> > > > > > > > > unchanged
> > > > > > > > > > > and
> > > > > > > > > > > > save initialized catalogs.This means that deletion
> > > > > > operations
> > > > > > > > from
> > > > > > > > > > one
> > > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > > 6.
> > > > > > > > > > > >
> > > > > > > > > > > > To support multi-session synchronization scenarios
> > for
> > > > > > > deletions
> > > > > > > > > > later
> > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > > configurable.There
> > > > > > > > > may
> > > > > > > > > > > be
> > > > > > > > > > > > three possible situations:
> > > > > > > > > > > >
> > > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > > >
> > > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > > automatically
> > > > > > > clear
> > > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > > >
> > > > > > > > > > > > c.No caching of any instances; each call to
> > getCatalog
> > > > > > > creates a
> > > > > > > > > new
> > > > > > > > > > > > instance.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > > >
> > > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you very much for your attention and
> suggestions
> > on
> > > > this
> > > > > > > > > FLIP. A
> > > > > > > > > > > > special thanks to Hang Ruan for his suggestions on
> the
> > > > entire
> > > > > > > > design
> > > > > > > > > > and
> > > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > > >
> > > > > > > > > > > > If you have any further suggestions or feedback about
> > > this
> > > > FLIP
> > > > > > > > > please
> > > > > > > > > > > feel
> > > > > > > > > > > > free to share.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > >
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for improving the FLIP. It looks good to me.
> > We
> > > > could
> > > > > > > > still
> > > > > > > > > > > > > reconsider in the future how to provide more common
> > > > built-in
> > > > > > > > cache
> > > > > > > > > > > > > functionality in CatalogManager so that not every
> > > > > > CatalogSotre
> > > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > > jinfeng1995@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There might be more such issues. I would
> suggest
> > > you
> > > > > > > > > completely
> > > > > > > > > > > walk
> > > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am very sorry for my carelessness and at the
> same
> > > > time, I
> > > > > > > > > greatly
> > > > > > > > > > > > > > appreciate your careful review. I have thoroughly
> > > > checked
> > > > > > the
> > > > > > > > > > entire
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> > > design,
> > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > could work without Optional CatalogStore being
> > > > configured.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, in the original design, CatalogStore was not
> > > > necessary
> > > > > > > > > because
> > > > > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs
> > to
> > > > store
> > > > > > > > > catalog
> > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > However, this caused inconsistency issues.
> > > Therefore, I
> > > > > > > > modified
> > > > > > > > > > this
> > > > > > > > > > > > > part
> > > > > > > > > > > > > > of the design and removed Map<String, Catalog>
> > > catalogs
> > > > > > from
> > > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > > At the same time, InMemoryCatalog will serve as
> > the
> > > > > > default
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > to save catalogs in memory and replace the
> > > > functionality of
> > > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > > The previous plan that kept
> > > > Map<String,Catalog>catalogs has
> > > > > > > > been
> > > > > > > > > > > moved
> > > > > > > > > > > > to
> > > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are still many places contain
> inconsistent
> > > > content,
> > > > > > > > e.g.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > > > 2. The java comment of the method
> > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not
> tell
> > > > what the
> > > > > > > > > method
> > > > > > > > > > is
> > > > > > > > > > > > > > doing.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There might be more such issues. I would
> suggest
> > > you
> > > > > > > > completely
> > > > > > > > > > > walk
> > > > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean
> that
> > > you
> > > > will
> > > > > > > > build
> > > > > > > > > > the
> > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > functionality in the CatalogStore? This is a
> very
> > > > > > different
> > > > > > > > > > design
> > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > from what the current FLIP described. If I am
> not
> > > > > > mistaken,
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > current FLIP design, CatalogManager could work
> > > > without
> > > > > > > > Optional
> > > > > > > > > > > > > > > CatalogStore being configured. That is the
> reason
> > > > why I
> > > > > > > > > mentioned
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > last email that the example code wrt the
> Optional
> > > > > > > > CatalogStore
> > > > > > > > > is
> > > > > > > > > > > not
> > > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you very much for the careful review. I
> > > have
> > > > > > > already
> > > > > > > > > made
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > multi-instance
> > > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > > > scenarios
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > current
> > > > > > > > > > > > > > > > design. I am thinking whether we should
> remove
> > > > > > > 'Map<String,
> > > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > > InMemoryCatalogStore.
> > > > > > > > This
> > > > > > > > > > > way,
> > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > can implement their own CatalogStore to
> achieve
> > > > > > > > > multi-instance
> > > > > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There are
> > still
> > > > some
> > > > > > > NIT
> > > > > > > > > > issues
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has
> > been
> > > > used
> > > > > > as
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > instead of Optional in the code example. It
> > > > should be
> > > > > > > > fine
> > > > > > > > > to
> > > > > > > > > > > use
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > pseudo code for now and update it after you
> > > > submit
> > > > > > the
> > > > > > > > PR.
> > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere
> in
> > > the
> > > > > > > > rejected
> > > > > > > > > > > > section
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > > > consistent.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > multi-instance
> > > > > > > > > > > scenarios,
> > > > > > > > > > > > I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > > > > Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > used as a cache, it should be invalid, once
> > the
> > > > > > related
> > > > > > > > > > catalog
> > > > > > > > > > > > has
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > removed from the CatalogStore by another
> > > > instance.
> > > > > > Did
> > > > > > > I
> > > > > > > > > miss
> > > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > > I think this is a good function name, I
> > have
> > > > > > updated
> > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Some common cache features should be
> > > > > > implemented
> > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> > alternative
> > > 1
> > > > > > turns
> > > > > > > > out
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > appropriate later, I will improve this
> part
> > > of
> > > > the
> > > > > > > > > content.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > As the above discussion moves forward,
> > the
> > > > > > option 2
> > > > > > > > > > > solution
> > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
> offline,
> > > we
> > > > > > think
> > > > > > > > that
> > > > > > > > > > > > > solution
> > > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > > inconsistency
> > > > > > > > issues.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There are some inconsistent
> descriptions
> > in
> > > > the
> > > > > > > > > content.
> > > > > > > > > > > > Would
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > > I will do my best to improve the document
> > and
> > > > > > > > appreciate
> > > > > > > > > > your
> > > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > can you put the unselected option in
> > > > `Rejected
> > > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > > > Alternatives`.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon
> FY
> > <
> > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I found there are two options in
> > `Proposed
> > > > > > > Changes`,
> > > > > > > > > can
> > > > > > > > > > > you
> > > > > > > > > > > > > put
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > > Alternatives`? I
> > > > > > > think
> > > > > > > > > > this
> > > > > > > > > > > > may
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am
> > still
> > > > > > confused
> > > > > > > > > after
> > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
> registerCatalog()
> > > and
> > > > > > > > > > addCatalog()
> > > > > > > > > > > > have
> > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > > difference based on their names. One
> of
> > > > them is
> > > > > > > > > > > responsible
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > persistence. How about
> > persistCatalog()?
> > > > > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > is
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > > persistence.
> > > > > > I
> > > > > > > > > would
> > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > > their purpose conceptually and
> clearly
> > in
> > > > the
> > > > > > > FLIP.
> > > > > > > > > > Some
> > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > features should be implemented, i.e.
> > data
> > > > in
> > > > > > the
> > > > > > > > > cache
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> > > > should be
> > > > > > > > found
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the cache(either it has been
> > initialized
> > > > or it
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > > consistency
> > > > will
> > > > > > > be
> > > > > > > > > > taken
> > > > > > > > > > > > care
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> > forward,
> > > > the
> > > > > > > > option
> > > > > > > > > 2
> > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> > because,
> > > > afaiu,
> > > > > > > > > issues
> > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > previously with option 1 are not
> solved
> > > > yet. Do
> > > > > > > you
> > > > > > > > > > still
> > > > > > > > > > > > > want
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > > both options and ask for suggestions
> > for
> > > > both
> > > > > > of
> > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there
> > are
> > > > some
> > > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the content. Would you like to clean
> > > them
> > > > up?
> > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng
> > Jin <
> > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you for your response, and I
> > > > completely
> > > > > > > > agree
> > > > > > > > > > > with
> > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > > Initially, I may have over
> > complicated
> > > > the
> > > > > > > whole
> > > > > > > > > > issue.
> > > > > > > > > > > > > First
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > > we need to consider the persistence
> > of
> > > > the
> > > > > > > > > Catalog's
> > > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > > If we only need to provide
> > persistence
> > > > for
> > > > > > > > Catalog
> > > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> > > > Catalog
> > > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > > > instance
> > > > > > to a
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > properties, and the default
> > > > implementation
> > > > > > will
> > > > > > > > > throw
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > > > properties
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > > * @return a map containing the
> > > > properties
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > > * @throws
> > > > UnsupportedOperationException if
> > > > > > > the
> > > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > > * the default implementation of
> > this
> > > > > > method
> > > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > > toProperties()
> > > > > > {
> > > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > > > CatalogStore,
> > > > > > > > the
> > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > will be called when registering a
> > > Catalog
> > > > > > > > instance
> > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > > catalog).
> > > > > > > > > > > The
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> > > > String>
> > > > > > > > > properties
> > > > > > > > > > > and
> > > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> > instances
> > > > do
> > > > > > not
> > > > > > > > > > > implement
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > > CatalogStore,
> > > > > > > the
> > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > > consistency
> > > > with
> > > > > > > the
> > > > > > > > > > > > original
> > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> > Catalog>
> > > > > > > catalogs
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > the same time can also avoid
> > conflicts
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
> itself
> > > and
> > > > > > > provide
> > > > > > > > a
> > > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > implementation for lazy loading,
> such
> > > as
> > > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > > > implements
> > > > > > > > > Catalog
> > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> > > String>
> > > > > > > > > properties)
> > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I have added this design to the
> FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> > Shammon
> > > > FY <
> > > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > > functionality
> > > > > > > of
> > > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > > > > `registerCatalog`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> > implementation
> > > in
> > > > > > Trino
> > > > > > > > is
> > > > > > > > > > also
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only
> has
> > > one
> > > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > catalog to memory and
> > `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> > another
> > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> > > Catalog>
> > > > > > > > catalogs`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> > > > before,
> > > > > > the
> > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > > `CatalogManager`.
> > > > > > How
> > > > > > > > > about
> > > > > > > > > > > > > saving
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs`
> and
> > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM
> Feng
> > > > Jin <
> > > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am
> > very
> > > > sorry
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
> original
> > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > pre-discussion
> > > > [1]
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > > > discussion
> > > > > > > > before.
> > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > whole solution, I found that
> the
> > > > logic of
> > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > need to change much. *We cannot
> > > only
> > > > > > > persist
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> > > > because
> > > > > > in
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > > we
> > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > the configurations
> corresponding
> > to
> > > > the
> > > > > > > > Catalog
> > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > > CatalogStore
> > > > > > > > interface
> > > > > > > > > > for
> > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> > part, I
> > > > also
> > > > > > > took
> > > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > > > catalog
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I
> > > will
> > > > > > > provide
> > > > > > > > a
> > > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > code examples for this part,
> and
> > > add
> > > > it
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > `addCatalog`
> > > > > > > still
> > > > > > > > > > looks
> > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > catalog
> > > to
> > > > > > > > > > CatalogStore,
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > > personal
> > > > > > > > opinion,
> > > > > > > > > > you
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
> intuitive
> > > to
> > > > me.
> > > > > > I
> > > > > > > > > don't
> > > > > > > > > > > see
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
> statement,
> > > > which
> > > > > > > > method
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > `addCatalog`?
> > > > I'm
> > > > > > > > > wondering
> > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > stement.
> > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> > > > design
> > > > > > was
> > > > > > > to
> > > > > > > > > add
> > > > > > > > > > > it
> > > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> > > disrupt
> > > > the
> > > > > > > > > existing
> > > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > > configuration to
> > > > > > > the
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the
> same
> > > time
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> > > > provide a
> > > > > > > > > default
> > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > interface
> `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> > otherwise
> > > we
> > > > > > would
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > > Map to store the configuration
> > for
> > > > lazy
> > > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > registration
> > > > for
> > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > I don't think
> > > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> > interface
> > > > > > because
> > > > > > > > > > Catalog
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > initialization
> > > > > > > for
> > > > > > > > > > > catalog
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > > inaccurate. It
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > > registration. I
> > > > > > > have
> > > > > > > > > > > already
> > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> > > > Shammon
> > > > > > FY <
> > > > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > > `registerCatalog`
> > > > > > > > > method
> > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
> method
> > > > named
> > > > > > > > > > `addCatalog`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> > memory
> > > > > > catalog
> > > > > > > > is a
> > > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> > exposed
> > > to
> > > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > pre-discussion
> > > > [1]
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
> conclusion
> > > of
> > > > that
> > > > > > > > > > > discussion
> > > > > > > > > > > > is
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> > interface
> > > > and
> > > > > > > > > implement
> > > > > > > > > > it
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > as memory, file and external
> > > > system.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
> between
> > > the
> > > > > > > current
> > > > > > > > > FLIP
> > > > > > > > > > > > > design
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > > What about the proposal of
> the
> > > > > > discussion
> > > > > > > > in
> > > > > > > > > > > thread
> > > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > >
> > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> 3:41 PM
> > > > yuxia <
> > > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving
> this
> > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> handling
> > > > catalog
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
> it'll
> > be
> > > > > > better
> > > > > > > to
> > > > > > > > > > > explain
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > these methods and give a
> > brief
> > > > > > example
> > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > `addCatalog`
> > > > > > > > still
> > > > > > > > > > > looks
> > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > > catalog to
> > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> very
> > > > personal
> > > > > > > > > > opinion,
> > > > > > > > > > > > you
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > > statement,
> > > > > > which
> > > > > > > > > method
> > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > `addCatalog`?
> > > > > > I'm
> > > > > > > > > > > wondering
> > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary
> to
> > > > > > provide a
> > > > > > > > > > default
> > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > registration
> > > > > > for
> > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog
> > with
> > > > CREATE
> > > > > > > > > > CATALOG,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> > by
> > > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
> registration
> > is
> > > > the
> > > > > > > > default
> > > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > > synchronous
> > > > > > > > > > > registration
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > > Will both method
> `addCatalog`
> > > and
> > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > > registration,
> > > > > > it
> > > > > > > > may
> > > > > > > > > > well
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > > executes successfully, but
> > then
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail
> to
> > > > open.
> > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break change
> > > which
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > > > > registration, I
> > > > > > > > > > > would
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to open
> > or
> > > > > > register
> > > > > > > > > > catalog
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will
> be
> > > > > > > initialized
> > > > > > > > > if
> > > > > > > > > > it
> > > > > > > > > > > > has
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> > > > process
> > > > > > > fails,
> > > > > > > > > > these
> > > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > initialization
> > > > > > > > for
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > > jinfeng1995@gmail.com
> > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > > dev@flink.apache.org
> > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> > > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> > > > Support
> > > > > > > > > > persistence
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
> registration
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all
> > for
> > > > your
> > > > > > > > > interest
> > > > > > > > > > in
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog
> > with
> > > > the
> > > > > > same
> > > > > > > > > name
> > > > > > > > > > > that
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> > > > point.
> > > > > > > Based
> > > > > > > > on
> > > > > > > > > > my
> > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs
> and
> > > > have
> > > > > > the
> > > > > > > > > > highest
> > > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > There are three methods
> that
> > > can
> > > > > > affect
> > > > > > > > the
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
> catalog
> > > > instance
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > catalog
> > > > > > > instance
> > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
> Catalog>
> > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method first retrieves
> > the
> > > > > > > > > corresponding
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > > > catalogs. If
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > > > > configuration
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts
> the
> > > > > > > initialized
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The following two methods
> > only
> > > > modify
> > > > > > > the
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > > catalogName,
> > > > > > > > > Map<String,
> > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > > properties
> > > > to
> > > > > > the
> > > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> > catalogName
> > > > with
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > specified
> > > > > > > > > > configuration
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The following are possible
> > > > conflict
> > > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > > > > Map<String,
> > > > > > > > > > > Catalog>,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly
> saved
> > > to
> > > > the
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > > Map<String,
> > > > > > > > Catalog>,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
> catalog)
> > > > method
> > > > > > > will
> > > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > String> properties) method
> > can
> > > > > > directly
> > > > > > > > > save
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > > getCatalog(String
> > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > > > > initialization
> > > > > > > > > > > because
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > catalog instance already
> > exists
> > > > in
> > > > > > > > catalogs
> > > > > > > > > > and
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > > corresponding to
> > > > > > > the
> > > > > > > > > > > original
> > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse
> > users
> > > > that
> > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
> catalog)`
> > > in
> > > > the
> > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > catalogName,
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This could potentially lead
> > to
> > > > > > > > confusion. I
> > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > > addCatalog(String
> > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > > previously
> > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > > add
> `registerCatalog(String
> > > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is already
> an
> > > > > > instance,
> > > > > > > > > > adding
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > > registerCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> > > > result in
> > > > > > > lazy
> > > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > > necessary,
> > > > but
> > > > > > > > > perhaps
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > > should be implemented in
> the
> > > > > > > CatalogStore
> > > > > > > > > > when
> > > > > > > > > > > > the
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > > > FileCatalogStore
> > > > > > > > > > could
> > > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
> while
> > > the
> > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary to
> > > have a
> > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> > > > Catalog
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > > > implement
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > > additional
> > > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > > registration
> > > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > > concern.
> > > > > > When
> > > > > > > > > > > executing
> > > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > > statements that reference a
> > > > specific
> > > > > > > > > Catalog,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > > will be initialized if it
> has
> > > not
> > > > > > been
> > > > > > > > > > > > initialized
> > > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > initialization process
> fails,
> > > > these
> > > > > > > > > > statements
> > > > > > > > > > > > will
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> > Catalog>
> > > > > > catalogs
> > > > > > > > and
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have
> > the
> > > > same
> > > > > > > > > catalog
> > > > > > > > > > > > name,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> instances.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This issue can be resolved
> by
> > > > > > referring
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> > > Catalog
> > > > > > that
> > > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > > will be given priority for
> > > > subsequent
> > > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> > 10:58 PM
> > > > > > Xianxun
> > > > > > > > Ye
> > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this
> > Flip,
> > > > I do
> > > > > > > > > believe
> > > > > > > > > > > this
> > > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I work
> > for
> > > > also
> > > > > > > > > manages a
> > > > > > > > > > > lot
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes
> > > slow
> > > > > > > because
> > > > > > > > of
> > > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in
> the
> > > > user's
> > > > > > > > Flink
> > > > > > > > > > SQL
> > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
> catalogs
> > > to
> > > > > > > improve
> > > > > > > > > the
> > > > > > > > > > > > speed
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I
> > have
> > > > some
> > > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
> catalog
> > > with
> > > > > > > CREATE
> > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> > by
> > > > > > default.
> > > > > > > > > What
> > > > > > > > > > if
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
> implement
> > > > > > > asynchronous
> > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > > > question
> > > > > > with
> > > > > > > > > > > Shammon’s
> > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> catalogs
> > > and
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the
> > > same
> > > > > > > catalog
> > > > > > > > > > name,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> instances.
> > > For
> > > > > > > example:
> > > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > > > hiveCatalog
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> > define
> > > > > > > `catalogs`
> > > > > > > > > and
> > > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang
> > Ruan
> > > <
> > > > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jark Wu <im...@gmail.com>.
Hi Jing,
Thank you for the update.
1. Could you move the CatalogStore registration API to the "Public
Interface" section?
"Proposed Changes" is more like a place to describe the implementation
details.
2. We should prefix "table." for the CatalogStore configuration. Besides,
the config key
name should be hierarchical[1]. Therefore, it may be better to use:
"table.catalog-store.kind"
"table.catalog-store.file.path"
3. I think Hang's suggestions make sense.
Others look good to me.
Best,
Jark
On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com> wrote:
> Hi, Feng.
>
> Thanks for the update.
> The current design looks good to me. I have some minor comments.
>
> 1. The `CatalogStore` need the `open`/`close` methods to init or close the
> resource. For example, when storing the information in MySQL, the store
> needs to open and close the connections.
>
> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>
> 3. About the usage in the sql gateway.
> Considering the usage in sql gateway, the sql gateway may create a
> CatalogStore for each session.
> If we are using the MySqlCatalogStore, there would be so many connections.
> How can we reuse the connection among these sessions?
> I think sql gateway need to maintain a connection pool in
> the CatalogStoreFactory and each session get its own connection from the
> pool when it is created.
> Then the `CatalogStoreFactory` may need the `open`/`close` methods to init
> or close its resource.
> Or is there a better way?
>
> Best,
> Hang
>
> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
>
> > Thanks Jingsong.
> >
> > > Just naming, maybe `createCatalog` in TableEnv
> >
> > +1 For this, I have already updated FLIP.
> >
> >
> > Best,
> > Feng
> >
> >
> > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com>
> > wrote:
> >
> > > Thanks Feng,
> > >
> > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > functions are converted to createxxx from registerxxx.
> > >
> > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com>
> wrote:
> > > >
> > > > Hi jark, thanks for your suggestion.
> > > >
> > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > CatalogStore should be immutable once TableEnv is created. Otherwise,
> > > there
> > > > might be some data inconsistencies when CatalogStore is changed.
> > > >
> > > > Yes, We should initialize the CatalogStore when creating the
> TableEnv.
> > > > Therefore, my current proposal is to add a method to configure the
> > > > CatalogStore in EnvironmentSettings.
> > > >
> > > > // Initialize a catalog Store
> > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > >
> > > > // set up the Table API
> > > > final EnvironmentSettings settings =
> > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > .withCatalogStore(catalogStore)
> > > > .build();
> > > >
> > > > final TableEnvironment tableEnv =
> > > TableEnvironment.create(settings);
> > > >
> > > >
> > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > method,
> > > > not an interface method?
> > > >
> > > > Sorry, While I did refer to the Catalog interface, I agree that as a
> > new
> > > > interface, the CatalogStoreFactory should not have a default method
> but
> > > an
> > > > interface method. I have already modified the interface.
> > > >
> > > > > 3. Please mention the alternative API in Javadoc for the
> > > > deprecated`registerCatalog`.
> > > > > 4. In the "Compatibility" section, would be better to mention the
> > > changed
> > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > persisted catalog store) is used.
> > > >
> > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > >
> > > >
> > > > [1].
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> > > >
> > > > > Hi Feng,
> > > > >
> > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > The current design looks pretty good to me. I just have some minor
> > > > > comments.
> > > > >
> > > > > 1. How to register the CatalogStore for Table API? I think the
> > > CatalogStore
> > > > > should be immutable once TableEnv is created. Otherwise, there
> might
> > be
> > > > > some data inconsistencies when CatalogStore is changed.
> > > > >
> > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > method,
> > > > > not an interface method?
> > > > >
> > > > > 3. Please mention the alternative API in Javadoc for the deprecated
> > > > > `registerCatalog`.
> > > > >
> > > > > 4. In the "Compatibility" section, would be better to mention the
> > > changed
> > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > > persisted catalog store) is used.
> > > > >
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi ， thanks all for reviewing the flip.
> > > > > >
> > > > > > @Ron
> > > > > >
> > > > > > > Regarding the CatalogStoreFactory#createCatalogStore method,
> do
> > we
> > > > > need
> > > > > > to provide a default implementation?
> > > > > >
> > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> > create
> > > an
> > > > > > InMemoryCatalogStore.
> > > > > >
> > > > > > > If we get a Catalog from CatalogStore, after initializing it,
> > > whether
> > > > > we
> > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > >
> > > > > > Yes, in the current design, catalogs are stored as snapshots, and
> > > once
> > > > > > initialized, the Catalog will be placed in the Map<String,
> Catalog>
> > > > > > catalogs.
> > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> primary
> > > > > source
> > > > > > for obtaining the corresponding Catalog.
> > > > > >
> > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > `catalog.store.path`?
> > > > > >
> > > > > > I think it is okay. Adding "sql" at the beginning may seem a bit
> > > > > strange. I
> > > > > > will update the FLIP.
> > > > > >
> > > > > >
> > > > > >
> > > > > > @Shammon
> > > > > >
> > > > > > Thank you for the review. I have made the necessary corrections.
> > > > > > Regarding the modifications made to the Public Interface, I have
> > also
> > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > >
> > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi feng,
> > > > > > >
> > > > > > > Thanks for updating, I have some minor comments
> > > > > > >
> > > > > > > 1. The modification of `CatalogManager` should not be in
> `Public
> > > > > > > Interfaces`, it is not a public interface.
> > > > > > >
> > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > > `CatalogStoreFactory`
> > > > > > >
> > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > catalogStore.get(catalogName);` in the `CatalogManager` should
> be
> > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > catalogStore.get(catalogName);`
> > > > > > >
> > > > > > > Best,
> > > > > > > Shammon FY
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi, Feng
> > > > > > > >
> > > > > > > > Thanks for driving this FLIP, this proposal is very useful
> for
> > > > > catalog
> > > > > > > > management.
> > > > > > > > I have some small questions:
> > > > > > > >
> > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> method,
> > > do we
> > > > > > > need
> > > > > > > > to provide a default implementation?
> > > > > > > > 2. If we get Catalog from CatalogStore, after initializing
> it,
> > > > > whether
> > > > > > we
> > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > > >
> > > > > > > > > Hi yuxia
> > > > > > > > >
> > > > > > > > > > But from the code in Proposed Changes, once we register
> > the
> > > > > > Catalog,
> > > > > > > > we
> > > > > > > > > initialize it and open it. right?
> > > > > > > > >
> > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> original
> > > > > CREATE
> > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > registerCatalog
> > > > > > so
> > > > > > > > > that parameter validation can be performed.
> > > > > > > > >
> > > > > > > > > In the current design, lazy initialization is mainly
> > reflected
> > > in
> > > > > > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > > > > > configurations,
> > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > luoyuxia@alumni.sjtu.edu.cn
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Feng.
> > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > initialization*.
> > > > > > If
> > > > > > > > i'm
> > > > > > > > > > wrong, please correct me.
> > > > > > > > > >
> > > > > > > > > > IIUC, lazy initialization means only you need to access
> the
> > > > > > catalog,
> > > > > > > > then
> > > > > > > > > > you initialize it. But from the code in Proposed Changes,
> > > once we
> > > > > > > > > register
> > > > > > > > > > the Catalog,
> > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Yuxia
> > > > > > > > > >
> > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> Catalog
> > > > > > > > configuration
> > > > > > > > > > and asynchronous registration
> > > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > >
> > > > > > > > > > One of the major changes is that current design will
> > provide
> > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> > > cache,
> > > > > > which
> > > > > > > > > means
> > > > > > > > > > once it has been initialized, any changes done by other
> > > sessions
> > > > > > will
> > > > > > > > not
> > > > > > > > > > affect it. Point 6 described follow-up options for
> further
> > > > > > > improvement.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > jinfeng1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all, I would like to update you on the latest
> progress
> > > of
> > > > > the
> > > > > > > > FLIP.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> > > ShengKai
> > > > > > Fang
> > > > > > > > > and I
> > > > > > > > > > > had an offline discussion regarding the overall
> solution
> > > for
> > > > > > Flink
> > > > > > > > > > > CatalogStore. We have reached a consensus and I have
> > > updated
> > > > > the
> > > > > > > > final
> > > > > > > > > > > solution in FLIP.
> > > > > > > > > > >
> > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > >
> > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > configuration
> > > > > > > similar
> > > > > > > > > to
> > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > 2.
> > > > > > > > > > >
> > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > storeCatalog(String
> > > > > > > > > > > catalogName, CatalogDescriptor) and
> CatalogDescriptor
> > > > > > > > > > getCatalog(String)
> > > > > > > > > > > will both use CatalogDescriptor instead of Catalog
> > > instance.
> > > > > > > This
> > > > > > > > > way,
> > > > > > > > > > > CatalogStore will only be responsible for saving and
> > > > > > retrieving
> > > > > > > > > > catalog
> > > > > > > > > > > configurations without having to initialize
> catalogs.
> > > > > > > > > > > 3.
> > > > > > > > > > >
> > > > > > > > > > > The default registerCatalog(String catalogName,
> > Catalog
> > > > > > catalog)
> > > > > > > > > > > function in CatalogManager will be marked as
> > deprecated.
> > > > > > > > > > > 4.
> > > > > > > > > > >
> > > > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > > > CatalogDescriptor
> > > > > > > > > > > catalog) will be added to serve as the default
> > > registration
> > > > > > > > function
> > > > > > > > > > for
> > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > 5.
> > > > > > > > > > >
> > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> > > remain
> > > > > > > > unchanged
> > > > > > > > > > and
> > > > > > > > > > > save initialized catalogs.This means that deletion
> > > > > operations
> > > > > > > from
> > > > > > > > > one
> > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > 6.
> > > > > > > > > > >
> > > > > > > > > > > To support multi-session synchronization scenarios
> for
> > > > > > deletions
> > > > > > > > > later
> > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > configurable.There
> > > > > > > > may
> > > > > > > > > > be
> > > > > > > > > > > three possible situations:
> > > > > > > > > > >
> > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > >
> > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > automatically
> > > > > > clear
> > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > >
> > > > > > > > > > > c.No caching of any instances; each call to
> getCatalog
> > > > > > creates a
> > > > > > > > new
> > > > > > > > > > > instance.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > >
> > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thank you very much for your attention and suggestions
> on
> > > this
> > > > > > > > FLIP. A
> > > > > > > > > > > special thanks to Hang Ruan for his suggestions on the
> > > entire
> > > > > > > design
> > > > > > > > > and
> > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > >
> > > > > > > > > > > If you have any further suggestions or feedback about
> > this
> > > FLIP
> > > > > > > > please
> > > > > > > > > > feel
> > > > > > > > > > > free to share.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for improving the FLIP. It looks good to me.
> We
> > > could
> > > > > > > still
> > > > > > > > > > > > reconsider in the future how to provide more common
> > > built-in
> > > > > > > cache
> > > > > > > > > > > > functionality in CatalogManager so that not every
> > > > > CatalogSotre
> > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > jinfeng1995@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > There might be more such issues. I would suggest
> > you
> > > > > > > > completely
> > > > > > > > > > walk
> > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am very sorry for my carelessness and at the same
> > > time, I
> > > > > > > > greatly
> > > > > > > > > > > > > appreciate your careful review. I have thoroughly
> > > checked
> > > > > the
> > > > > > > > > entire
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> > design,
> > > > > > > > > > CatalogManager
> > > > > > > > > > > > > could work without Optional CatalogStore being
> > > configured.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, in the original design, CatalogStore was not
> > > necessary
> > > > > > > > because
> > > > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs
> to
> > > store
> > > > > > > > catalog
> > > > > > > > > > > > > instances.
> > > > > > > > > > > > > However, this caused inconsistency issues.
> > Therefore, I
> > > > > > > modified
> > > > > > > > > this
> > > > > > > > > > > > part
> > > > > > > > > > > > > of the design and removed Map<String, Catalog>
> > catalogs
> > > > > from
> > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > At the same time, InMemoryCatalog will serve as
> the
> > > > > default
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > to save catalogs in memory and replace the
> > > functionality of
> > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > The previous plan that kept
> > > Map<String,Catalog>catalogs has
> > > > > > > been
> > > > > > > > > > moved
> > > > > > > > > > > to
> > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There are still many places contain inconsistent
> > > content,
> > > > > > > e.g.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > > 2. The java comment of the method
> > > registerCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not tell
> > > what the
> > > > > > > > method
> > > > > > > > > is
> > > > > > > > > > > > > doing.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There might be more such issues. I would suggest
> > you
> > > > > > > completely
> > > > > > > > > > walk
> > > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean that
> > you
> > > will
> > > > > > > build
> > > > > > > > > the
> > > > > > > > > > > > cache
> > > > > > > > > > > > > > functionality in the CatalogStore? This is a very
> > > > > different
> > > > > > > > > design
> > > > > > > > > > > > > concept
> > > > > > > > > > > > > > from what the current FLIP described. If I am not
> > > > > mistaken,
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > current FLIP design, CatalogManager could work
> > > without
> > > > > > > Optional
> > > > > > > > > > > > > > CatalogStore being configured. That is the reason
> > > why I
> > > > > > > > mentioned
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > last email that the example code wrt the Optional
> > > > > > > CatalogStore
> > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you very much for the careful review. I
> > have
> > > > > > already
> > > > > > > > made
> > > > > > > > > > the
> > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > multi-instance
> > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > > scenarios
> > > > > with
> > > > > > > the
> > > > > > > > > > > current
> > > > > > > > > > > > > > > design. I am thinking whether we should remove
> > > > > > 'Map<String,
> > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > InMemoryCatalogStore.
> > > > > > > This
> > > > > > > > > > way,
> > > > > > > > > > > > > users
> > > > > > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > > > > > multi-instance
> > > > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for working on the FLIP. There are
> still
> > > some
> > > > > > NIT
> > > > > > > > > issues
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has
> been
> > > used
> > > > > as
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > instead of Optional in the code example. It
> > > should be
> > > > > > > fine
> > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > > > it
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > pseudo code for now and update it after you
> > > submit
> > > > > the
> > > > > > > PR.
> > > > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in
> > the
> > > > > > > rejected
> > > > > > > > > > > section
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > > consistent.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > multi-instance
> > > > > > > > > > scenarios,
> > > > > > > > > > > I
> > > > > > > > > > > > am
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > > > Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > used as a cache, it should be invalid, once
> the
> > > > > related
> > > > > > > > > catalog
> > > > > > > > > > > has
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > removed from the CatalogStore by another
> > > instance.
> > > > > Did
> > > > > > I
> > > > > > > > miss
> > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > I think this is a good function name, I
> have
> > > > > updated
> > > > > > it
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Some common cache features should be
> > > > > implemented
> > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> alternative
> > 1
> > > > > turns
> > > > > > > out
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > appropriate later, I will improve this part
> > of
> > > the
> > > > > > > > content.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As the above discussion moves forward,
> the
> > > > > option 2
> > > > > > > > > > solution
> > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > Yes, after discussing with Shammon offline,
> > we
> > > > > think
> > > > > > > that
> > > > > > > > > > > > solution
> > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > inconsistency
> > > > > > > issues.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There are some inconsistent descriptions
> in
> > > the
> > > > > > > > content.
> > > > > > > > > > > Would
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > I will do my best to improve the document
> and
> > > > > > > appreciate
> > > > > > > > > your
> > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > can you put the unselected option in
> > > `Rejected
> > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > > Alternatives`.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY
> <
> > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I found there are two options in
> `Proposed
> > > > > > Changes`,
> > > > > > > > can
> > > > > > > > > > you
> > > > > > > > > > > > put
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > Alternatives`? I
> > > > > > think
> > > > > > > > > this
> > > > > > > > > > > may
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am
> still
> > > > > confused
> > > > > > > > after
> > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog()
> > and
> > > > > > > > > addCatalog()
> > > > > > > > > > > have
> > > > > > > > > > > > > no
> > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > difference based on their names. One of
> > > them is
> > > > > > > > > > responsible
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > persistence. How about
> persistCatalog()?
> > > > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > is
> > > > > > > > > > > > used
> > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > persistence.
> > > > > I
> > > > > > > > would
> > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > their purpose conceptually and clearly
> in
> > > the
> > > > > > FLIP.
> > > > > > > > > Some
> > > > > > > > > > > > common
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > features should be implemented, i.e.
> data
> > > in
> > > > > the
> > > > > > > > cache
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> > > should be
> > > > > > > found
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > store
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the cache(either it has been
> initialized
> > > or it
> > > > > > will
> > > > > > > > be
> > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > consistency
> > > will
> > > > > > be
> > > > > > > > > taken
> > > > > > > > > > > care
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> forward,
> > > the
> > > > > > > option
> > > > > > > > 2
> > > > > > > > > > > > solution
> > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> because,
> > > afaiu,
> > > > > > > > issues
> > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > previously with option 1 are not solved
> > > yet. Do
> > > > > > you
> > > > > > > > > still
> > > > > > > > > > > > want
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > both options and ask for suggestions
> for
> > > both
> > > > > of
> > > > > > > > them?
> > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there
> are
> > > some
> > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the content. Would you like to clean
> > them
> > > up?
> > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng
> Jin <
> > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thank you for your response, and I
> > > completely
> > > > > > > agree
> > > > > > > > > > with
> > > > > > > > > > > > your
> > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > Initially, I may have over
> complicated
> > > the
> > > > > > whole
> > > > > > > > > issue.
> > > > > > > > > > > > First
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > we need to consider the persistence
> of
> > > the
> > > > > > > > Catalog's
> > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > If we only need to provide
> persistence
> > > for
> > > > > > > Catalog
> > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> > > Catalog
> > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > > instance
> > > > > to a
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > properties, and the default
> > > implementation
> > > > > will
> > > > > > > > throw
> > > > > > > > > > an
> > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > > properties
> > > > > of
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > * @return a map containing the
> > > properties
> > > > > of
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > * @throws
> > > UnsupportedOperationException if
> > > > > > the
> > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > * the default implementation of
> this
> > > > > method
> > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > toProperties()
> > > > > {
> > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > > CatalogStore,
> > > > > > > the
> > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > will be called when registering a
> > Catalog
> > > > > > > instance
> > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > Catalog
> > > > > > > > catalog).
> > > > > > > > > > The
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> > > String>
> > > > > > > > properties
> > > > > > > > > > and
> > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> instances
> > > do
> > > > > not
> > > > > > > > > > implement
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > CatalogStore,
> > > > > > the
> > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > consistency
> > > with
> > > > > > the
> > > > > > > > > > > original
> > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> Catalog>
> > > > > > catalogs
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > the same time can also avoid
> conflicts
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > we can start from the Catalog itself
> > and
> > > > > > provide
> > > > > > > a
> > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > implementation for lazy loading, such
> > as
> > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > > implements
> > > > > > > > Catalog
> > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> > String>
> > > > > > > > properties)
> > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> Shammon
> > > FY <
> > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > functionality
> > > > > > of
> > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > > > `registerCatalog`
> > > > > > > > > or
> > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> implementation
> > in
> > > > > Trino
> > > > > > > is
> > > > > > > > > also
> > > > > > > > > > > the
> > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only has
> > one
> > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > catalog to memory and
> `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> another
> > > > > > > > > > `registerCatalog`
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> > Catalog>
> > > > > > > catalogs`
> > > > > > > > > or
> > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> > > before,
> > > > > the
> > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > `CatalogManager`.
> > > > > How
> > > > > > > > about
> > > > > > > > > > > > saving
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > > > > > `CatalogStore
> > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng
> > > Jin <
> > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am
> very
> > > sorry
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > by my deviation from the original
> > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > I found there is a
> pre-discussion
> > > [1]
> > > > > for
> > > > > > > > this
> > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > > discussion
> > > > > > > before.
> > > > > > > > > > > > However,
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > whole solution, I found that the
> > > logic of
> > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > need to change much. *We cannot
> > only
> > > > > > persist
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > registerCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> > > because
> > > > > in
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > we
> > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > the configurations corresponding
> to
> > > the
> > > > > > > Catalog
> > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > CatalogStore
> > > > > > > interface
> > > > > > > > > for
> > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> part, I
> > > also
> > > > > > took
> > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > > catalog
> > > > > with
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > name
> > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I
> > will
> > > > > > provide
> > > > > > > a
> > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > code examples for this part, and
> > add
> > > it
> > > > > to
> > > > > > > the
> > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > `addCatalog`
> > > > > > still
> > > > > > > > > looks
> > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > this method is for storing
> catalog
> > to
> > > > > > > > > CatalogStore,
> > > > > > > > > > > how
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > personal
> > > > > > > opinion,
> > > > > > > > > you
> > > > > > > > > > > can
> > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive
> > to
> > > me.
> > > > > I
> > > > > > > > don't
> > > > > > > > > > see
> > > > > > > > > > > > any
> > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement,
> > > which
> > > > > > > method
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> `addCatalog`?
> > > I'm
> > > > > > > > wondering
> > > > > > > > > > > > whether
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> stement.
> > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> > > design
> > > > > was
> > > > > > to
> > > > > > > > add
> > > > > > > > > > it
> > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> > disrupt
> > > the
> > > > > > > > existing
> > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > configuration to
> > > > > > the
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the same
> > time
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> > > provide a
> > > > > > > > default
> > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> otherwise
> > we
> > > > > would
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > Map to store the configuration
> for
> > > lazy
> > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > registration
> > > for
> > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > I don't think
> > registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> interface
> > > > > because
> > > > > > > > > Catalog
> > > > > > > > > > > is
> > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > initialization
> > > > > > for
> > > > > > > > > > catalog
> > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > inaccurate. It
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > registration. I
> > > > > > have
> > > > > > > > > > already
> > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> > > Shammon
> > > > > FY <
> > > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > `registerCatalog`
> > > > > > > > method
> > > > > > > > > in
> > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > confuse users whether a method
> > > named
> > > > > > > > > `addCatalog`
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> memory
> > > > > catalog
> > > > > > > is a
> > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> exposed
> > to
> > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I found there is a
> pre-discussion
> > > [1]
> > > > > for
> > > > > > > > this
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion
> > of
> > > that
> > > > > > > > > > discussion
> > > > > > > > > > > is
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> interface
> > > and
> > > > > > > > implement
> > > > > > > > > it
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > as memory, file and external
> > > system.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think there is a gap between
> > the
> > > > > > current
> > > > > > > > FLIP
> > > > > > > > > > > > design
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > What about the proposal of the
> > > > > discussion
> > > > > > > in
> > > > > > > > > > thread
> > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > >
> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM
> > > yuxia <
> > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this
> > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > > catalog
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think it'll
> be
> > > > > better
> > > > > > to
> > > > > > > > > > explain
> > > > > > > > > > > > it
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > these methods and give a
> brief
> > > > > example
> > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > `addCatalog`
> > > > > > > still
> > > > > > > > > > looks
> > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > catalog to
> > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > > personal
> > > > > > > > > opinion,
> > > > > > > > > > > you
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > statement,
> > > > > which
> > > > > > > > method
> > > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > `addCatalog`?
> > > > > I'm
> > > > > > > > > > wondering
> > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > stement.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> > > > > provide a
> > > > > > > > > default
> > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > interface
> > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > registration
> > > > > for
> > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog
> with
> > > CREATE
> > > > > > > > > CATALOG,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> by
> > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > If asynchronous registration
> is
> > > the
> > > > > > > default
> > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > synchronous
> > > > > > > > > > registration
> > > > > > > > > > > > just
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > Will both method `addCatalog`
> > and
> > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > registration,
> > > > > it
> > > > > > > may
> > > > > > > > > well
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > executes successfully, but
> then
> > > the
> > > > > > > > following
> > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail to
> > > open.
> > > > > > > > > > > > > > > > > > > > > > > > I think it's a break change
> > which
> > > > > > should
> > > > > > > be
> > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > > > registration, I
> > > > > > > > > > would
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > will be an executor to open
> or
> > > > > register
> > > > > > > > > catalog
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > > > > > initialized
> > > > > > > > if
> > > > > > > > > it
> > > > > > > > > > > has
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> > > process
> > > > > > fails,
> > > > > > > > > these
> > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > initialization
> > > > > > > for
> > > > > > > > > > > catalog
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > jinfeng1995@gmail.com
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > dev@flink.apache.org
> > > >
> > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> > > Support
> > > > > > > > > persistence
> > > > > > > > > > > of
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all
> for
> > > your
> > > > > > > > interest
> > > > > > > > > in
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog
> with
> > > the
> > > > > same
> > > > > > > > name
> > > > > > > > > > that
> > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> > > point.
> > > > > > Based
> > > > > > > on
> > > > > > > > > my
> > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs and
> > > have
> > > > > the
> > > > > > > > > highest
> > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > There are three methods that
> > can
> > > > > affect
> > > > > > > the
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This method puts the catalog
> > > instance
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> catalog
> > > > > > instance
> > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This method first retrieves
> the
> > > > > > > > corresponding
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > > catalogs. If
> > > > > > the
> > > > > > > > > > catalog
> > > > > > > > > > > > > does
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > > > configuration
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > > > > > initialized
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> catalogs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The following two methods
> only
> > > modify
> > > > > > the
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > catalogName,
> > > > > > > > Map<String,
> > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > properties
> > > to
> > > > > the
> > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> catalogName
> > > with
> > > > > the
> > > > > > > > same
> > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > specified
> > > > > > > > > configuration
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The following are possible
> > > conflict
> > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > > catalogName
> > > > > > > > already
> > > > > > > > > > > > exists
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > > > Map<String,
> > > > > > > > > > Catalog>,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > catalogName,
> > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly saved
> > to
> > > the
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > > catalogName
> > > > > > > > already
> > > > > > > > > > > > exists
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > Map<String,
> > > > > > > Catalog>,
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog)
> > > method
> > > > > > will
> > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > > catalogName
> > > > > > > > already
> > > > > > > > > > > > exists
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > > addCatalog(String
> > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > String> properties) method
> can
> > > > > directly
> > > > > > > > save
> > > > > > > > > > the
> > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > getCatalog(String
> > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > > > initialization
> > > > > > > > > > because
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > catalog instance already
> exists
> > > in
> > > > > > > catalogs
> > > > > > > > > and
> > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > corresponding to
> > > > > > the
> > > > > > > > > > original
> > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse
> users
> > > that
> > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)`
> > in
> > > the
> > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > catalogName,
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This could potentially lead
> to
> > > > > > > confusion. I
> > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > addCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > previously
> > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> > > > > instance,
> > > > > > > > > adding
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > registerCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> > > result in
> > > > > > lazy
> > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > necessary,
> > > but
> > > > > > > > perhaps
> > > > > > > > > > the
> > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > should be implemented in the
> > > > > > CatalogStore
> > > > > > > > > when
> > > > > > > > > > > the
> > > > > > > > > > > > > data
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > > FileCatalogStore
> > > > > > > > > could
> > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file, while
> > the
> > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary to
> > have a
> > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> > > Catalog
> > > > > > > > > > > configurations
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > > implement
> > > > > > > > > asynchronous
> > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > additional
> > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > registration
> > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > concern.
> > > > > When
> > > > > > > > > > executing
> > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > statements that reference a
> > > specific
> > > > > > > > Catalog,
> > > > > > > > > > the
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > will be initialized if it has
> > not
> > > > > been
> > > > > > > > > > > initialized
> > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > initialization process fails,
> > > these
> > > > > > > > > statements
> > > > > > > > > > > will
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> Catalog>
> > > > > catalogs
> > > > > > > and
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have
> the
> > > same
> > > > > > > > catalog
> > > > > > > > > > > name,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> > > > > referring
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> > Catalog
> > > > > that
> > > > > > > has
> > > > > > > > > been
> > > > > > > > > > > > most
> > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > will be given priority for
> > > subsequent
> > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> 10:58 PM
> > > > > Xianxun
> > > > > > > Ye
> > > > > > > > <
> > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this
> Flip,
> > > I do
> > > > > > > > believe
> > > > > > > > > > this
> > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I work
> for
> > > also
> > > > > > > > manages a
> > > > > > > > > > lot
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes
> > slow
> > > > > > because
> > > > > > > of
> > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the
> > > user's
> > > > > > > Flink
> > > > > > > > > SQL
> > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs
> > to
> > > > > > improve
> > > > > > > > the
> > > > > > > > > > > speed
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I
> have
> > > some
> > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog
> > with
> > > > > > CREATE
> > > > > > > > > > CATALOG,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> by
> > > > > default.
> > > > > > > > What
> > > > > > > > > if
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > > > > > asynchronous
> > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > > question
> > > > > with
> > > > > > > > > > Shammon’s
> > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs
> > and
> > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the
> > same
> > > > > > catalog
> > > > > > > > > name,
> > > > > > > > > > > but
> > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > For
> > > > > > example:
> > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > > hiveCatalog
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> define
> > > > > > `catalogs`
> > > > > > > > and
> > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang
> Ruan
> > <
> > > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Feng.
Thanks for the update.
The current design looks good to me. I have some minor comments.
1. The `CatalogStore` need the `open`/`close` methods to init or close the
resource. For example, when storing the information in MySQL, the store
needs to open and close the connections.
2. The `getCatalog` is misspelled as `optionalDescriptor`.
3. About the usage in the sql gateway.
Considering the usage in sql gateway, the sql gateway may create a
CatalogStore for each session.
If we are using the MySqlCatalogStore, there would be so many connections.
How can we reuse the connection among these sessions?
I think sql gateway need to maintain a connection pool in
the CatalogStoreFactory and each session get its own connection from the
pool when it is created.
Then the `CatalogStoreFactory` may need the `open`/`close` methods to init
or close its resource.
Or is there a better way?
Best,
Hang
Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> Thanks Jingsong.
>
> > Just naming, maybe `createCatalog` in TableEnv
>
> +1 For this, I have already updated FLIP.
>
>
> Best,
> Feng
>
>
> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com>
> wrote:
>
> > Thanks Feng,
> >
> > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > functions are converted to createxxx from registerxxx.
> >
> > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com> wrote:
> > >
> > > Hi jark, thanks for your suggestion.
> > >
> > > > 1. How to register the CatalogStore for Table API? I think the
> > > CatalogStore should be immutable once TableEnv is created. Otherwise,
> > there
> > > might be some data inconsistencies when CatalogStore is changed.
> > >
> > > Yes, We should initialize the CatalogStore when creating the TableEnv.
> > > Therefore, my current proposal is to add a method to configure the
> > > CatalogStore in EnvironmentSettings.
> > >
> > > // Initialize a catalog Store
> > > CatalogStore catalogStore = new FileCatalogStore("");
> > >
> > > // set up the Table API
> > > final EnvironmentSettings settings =
> > > EnvironmentSettings.newInstance().inBatchMode()
> > > .withCatalogStore(catalogStore)
> > > .build();
> > >
> > > final TableEnvironment tableEnv =
> > TableEnvironment.create(settings);
> > >
> > >
> > > > 2. Why does the CatalogStoreFactory interface only have a default
> > method,
> > > not an interface method?
> > >
> > > Sorry, While I did refer to the Catalog interface, I agree that as a
> new
> > > interface, the CatalogStoreFactory should not have a default method but
> > an
> > > interface method. I have already modified the interface.
> > >
> > > > 3. Please mention the alternative API in Javadoc for the
> > > deprecated`registerCatalog`.
> > > > 4. In the "Compatibility" section, would be better to mention the
> > changed
> > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > persisted catalog store) is used.
> > >
> > > Thanks for the suggestion, I have updated the FLIP [1].
> > >
> > >
> > > [1].
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> > >
> > > > Hi Feng,
> > > >
> > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > The current design looks pretty good to me. I just have some minor
> > > > comments.
> > > >
> > > > 1. How to register the CatalogStore for Table API? I think the
> > CatalogStore
> > > > should be immutable once TableEnv is created. Otherwise, there might
> be
> > > > some data inconsistencies when CatalogStore is changed.
> > > >
> > > > 2. Why does the CatalogStoreFactory interface only have a default
> > method,
> > > > not an interface method?
> > > >
> > > > 3. Please mention the alternative API in Javadoc for the deprecated
> > > > `registerCatalog`.
> > > >
> > > > 4. In the "Compatibility" section, would be better to mention the
> > changed
> > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > persisted catalog store) is used.
> > > >
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
> > > >
> > > > > Hi ， thanks all for reviewing the flip.
> > > > >
> > > > > @Ron
> > > > >
> > > > > > Regarding the CatalogStoreFactory#createCatalogStore method, do
> we
> > > > need
> > > > > to provide a default implementation?
> > > > >
> > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> create
> > an
> > > > > InMemoryCatalogStore.
> > > > >
> > > > > > If we get a Catalog from CatalogStore, after initializing it,
> > whether
> > > > we
> > > > > put it in Map<String, Catalog> catalogs again?
> > > > >
> > > > > Yes, in the current design, catalogs are stored as snapshots, and
> > once
> > > > > initialized, the Catalog will be placed in the Map<String, Catalog>
> > > > > catalogs.
> > > > > Subsequently, the Map<String, Catalog> catalogs will be the primary
> > > > source
> > > > > for obtaining the corresponding Catalog.
> > > > >
> > > > > > how about renaming them to `catalog.store.type` and
> > > > > `catalog.store.path`?
> > > > >
> > > > > I think it is okay. Adding "sql" at the beginning may seem a bit
> > > > strange. I
> > > > > will update the FLIP.
> > > > >
> > > > >
> > > > >
> > > > > @Shammon
> > > > >
> > > > > Thank you for the review. I have made the necessary corrections.
> > > > > Regarding the modifications made to the Public Interface, I have
> also
> > > > > included the relevant changes to the `TableEnvironment`.
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > >
> > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi feng,
> > > > > >
> > > > > > Thanks for updating, I have some minor comments
> > > > > >
> > > > > > 1. The modification of `CatalogManager` should not be in `Public
> > > > > > Interfaces`, it is not a public interface.
> > > > > >
> > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > `CatalogStoreFactory`
> > > > > >
> > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > catalogStore.get(catalogName);`
> > > > > >
> > > > > > Best,
> > > > > > Shammon FY
> > > > > >
> > > > > >
> > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi, Feng
> > > > > > >
> > > > > > > Thanks for driving this FLIP, this proposal is very useful for
> > > > catalog
> > > > > > > management.
> > > > > > > I have some small questions:
> > > > > > >
> > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore method,
> > do we
> > > > > > need
> > > > > > > to provide a default implementation?
> > > > > > > 2. If we get Catalog from CatalogStore, after initializing it,
> > > > whether
> > > > > we
> > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > >
> > > > > > > Best,
> > > > > > > Ron
> > > > > > >
> > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > >
> > > > > > > > Hi yuxia
> > > > > > > >
> > > > > > > > > But from the code in Proposed Changes, once we register
> the
> > > > > Catalog,
> > > > > > > we
> > > > > > > > initialize it and open it. right?
> > > > > > > >
> > > > > > > > Yes, In order to avoid inconsistent semantics of the original
> > > > CREATE
> > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > registerCatalog
> > > > > so
> > > > > > > > that parameter validation can be performed.
> > > > > > > >
> > > > > > > > In the current design, lazy initialization is mainly
> reflected
> > in
> > > > > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > > > > configurations,
> > > > > > > > only initialization is required in getCatalog.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng
> > > > > > > >
> > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > luoyuxia@alumni.sjtu.edu.cn
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Feng.
> > > > > > > > > I'm trying to understanding the meaning of *lazy
> > initialization*.
> > > > > If
> > > > > > > i'm
> > > > > > > > > wrong, please correct me.
> > > > > > > > >
> > > > > > > > > IIUC, lazy initialization means only you need to access the
> > > > > catalog,
> > > > > > > then
> > > > > > > > > you initialize it. But from the code in Proposed Changes,
> > once we
> > > > > > > > register
> > > > > > > > > the Catalog,
> > > > > > > > > we initialize it and open it. right?
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Yuxia
> > > > > > > > >
> > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > > > configuration
> > > > > > > > > and asynchronous registration
> > > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > >
> > > > > > > > > One of the major changes is that current design will
> provide
> > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> > cache,
> > > > > which
> > > > > > > > means
> > > > > > > > > once it has been initialized, any changes done by other
> > sessions
> > > > > will
> > > > > > > not
> > > > > > > > > affect it. Point 6 described follow-up options for further
> > > > > > improvement.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > jinfeng1995@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all, I would like to update you on the latest progress
> > of
> > > > the
> > > > > > > FLIP.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> > ShengKai
> > > > > Fang
> > > > > > > > and I
> > > > > > > > > > had an offline discussion regarding the overall solution
> > for
> > > > > Flink
> > > > > > > > > > CatalogStore. We have reached a consensus and I have
> > updated
> > > > the
> > > > > > > final
> > > > > > > > > > solution in FLIP.
> > > > > > > > > >
> > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > configuration
> > > > > > similar
> > > > > > > > to
> > > > > > > > > > TableDescriptor.
> > > > > > > > > > 2.
> > > > > > > > > >
> > > > > > > > > > The two key functions of CatalogStore - void
> > > > > storeCatalog(String
> > > > > > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > > > > > getCatalog(String)
> > > > > > > > > > will both use CatalogDescriptor instead of Catalog
> > instance.
> > > > > > This
> > > > > > > > way,
> > > > > > > > > > CatalogStore will only be responsible for saving and
> > > > > retrieving
> > > > > > > > > catalog
> > > > > > > > > > configurations without having to initialize catalogs.
> > > > > > > > > > 3.
> > > > > > > > > >
> > > > > > > > > > The default registerCatalog(String catalogName,
> Catalog
> > > > > catalog)
> > > > > > > > > > function in CatalogManager will be marked as
> deprecated.
> > > > > > > > > > 4.
> > > > > > > > > >
> > > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > > CatalogDescriptor
> > > > > > > > > > catalog) will be added to serve as the default
> > registration
> > > > > > > function
> > > > > > > > > for
> > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > 5.
> > > > > > > > > >
> > > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> > remain
> > > > > > > unchanged
> > > > > > > > > and
> > > > > > > > > > save initialized catalogs.This means that deletion
> > > > operations
> > > > > > from
> > > > > > > > one
> > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > 6.
> > > > > > > > > >
> > > > > > > > > > To support multi-session synchronization scenarios for
> > > > > deletions
> > > > > > > > later
> > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > configurable.There
> > > > > > > may
> > > > > > > > > be
> > > > > > > > > > three possible situations:
> > > > > > > > > >
> > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > >
> > > > > > > > > > b.Introduction of LRU cache logic which can
> > automatically
> > > > > clear
> > > > > > > > > > long-unused catalogs.
> > > > > > > > > >
> > > > > > > > > > c.No caching of any instances; each call to getCatalog
> > > > > creates a
> > > > > > > new
> > > > > > > > > > instance.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This is the document for discussion:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > >
> > > > > > > > > > This is the final proposal document:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thank you very much for your attention and suggestions on
> > this
> > > > > > > FLIP. A
> > > > > > > > > > special thanks to Hang Ruan for his suggestions on the
> > entire
> > > > > > design
> > > > > > > > and
> > > > > > > > > > organizing offline discussions.
> > > > > > > > > >
> > > > > > > > > > If you have any further suggestions or feedback about
> this
> > FLIP
> > > > > > > please
> > > > > > > > > feel
> > > > > > > > > > free to share.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for improving the FLIP. It looks good to me. We
> > could
> > > > > > still
> > > > > > > > > > > reconsider in the future how to provide more common
> > built-in
> > > > > > cache
> > > > > > > > > > > functionality in CatalogManager so that not every
> > > > CatalogSotre
> > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > jinfeng1995@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > >
> > > > > > > > > > > > > There might be more such issues. I would suggest
> you
> > > > > > > completely
> > > > > > > > > walk
> > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > >
> > > > > > > > > > > > I am very sorry for my carelessness and at the same
> > time, I
> > > > > > > greatly
> > > > > > > > > > > > appreciate your careful review. I have thoroughly
> > checked
> > > > the
> > > > > > > > entire
> > > > > > > > > > FLIP
> > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> design,
> > > > > > > > > CatalogManager
> > > > > > > > > > > > could work without Optional CatalogStore being
> > configured.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, in the original design, CatalogStore was not
> > necessary
> > > > > > > because
> > > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs to
> > store
> > > > > > > catalog
> > > > > > > > > > > > instances.
> > > > > > > > > > > > However, this caused inconsistency issues.
> Therefore, I
> > > > > > modified
> > > > > > > > this
> > > > > > > > > > > part
> > > > > > > > > > > > of the design and removed Map<String, Catalog>
> catalogs
> > > > from
> > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > At the same time, InMemoryCatalog will serve as the
> > > > default
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > to save catalogs in memory and replace the
> > functionality of
> > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > The previous plan that kept
> > Map<String,Catalog>catalogs has
> > > > > > been
> > > > > > > > > moved
> > > > > > > > > > to
> > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > There are still many places contain inconsistent
> > content,
> > > > > > e.g.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > 2. The java comment of the method
> > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog catalog) in CatalogManager does not tell
> > what the
> > > > > > > method
> > > > > > > > is
> > > > > > > > > > > > doing.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > There might be more such issues. I would suggest
> you
> > > > > > completely
> > > > > > > > > walk
> > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > >
> > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean that
> you
> > will
> > > > > > build
> > > > > > > > the
> > > > > > > > > > > cache
> > > > > > > > > > > > > functionality in the CatalogStore? This is a very
> > > > different
> > > > > > > > design
> > > > > > > > > > > > concept
> > > > > > > > > > > > > from what the current FLIP described. If I am not
> > > > mistaken,
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > current FLIP design, CatalogManager could work
> > without
> > > > > > Optional
> > > > > > > > > > > > > CatalogStore being configured. That is the reason
> > why I
> > > > > > > mentioned
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > last email that the example code wrt the Optional
> > > > > > CatalogStore
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > > > > correct.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you very much for the careful review. I
> have
> > > > > already
> > > > > > > made
> > > > > > > > > the
> > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > multi-instance
> > > > > > > > > > scenarios, I
> > > > > > > > > > > > am
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > scenarios
> > > > with
> > > > > > the
> > > > > > > > > > current
> > > > > > > > > > > > > > design. I am thinking whether we should remove
> > > > > 'Map<String,
> > > > > > > > > > Catalog>'
> > > > > > > > > > > > and
> > > > > > > > > > > > > > make Cache the default behavior of
> > > > InMemoryCatalogStore.
> > > > > > This
> > > > > > > > > way,
> > > > > > > > > > > > users
> > > > > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > > > > multi-instance
> > > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for working on the FLIP. There are still
> > some
> > > > > NIT
> > > > > > > > issues
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been
> > used
> > > > as
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > instead of Optional in the code example. It
> > should be
> > > > > > fine
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > it
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > pseudo code for now and update it after you
> > submit
> > > > the
> > > > > > PR.
> > > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in
> the
> > > > > > rejected
> > > > > > > > > > section
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > consistent.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > multi-instance
> > > > > > > > > scenarios,
> > > > > > > > > > I
> > > > > > > > > > > am
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > used as a cache, it should be invalid, once the
> > > > related
> > > > > > > > catalog
> > > > > > > > > > has
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > removed from the CatalogStore by another
> > instance.
> > > > Did
> > > > > I
> > > > > > > miss
> > > > > > > > > > > > > something?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > I think this is a good function name, I have
> > > > updated
> > > > > it
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Some common cache features should be
> > > > implemented
> > > > > > > > > > > > > > > > Thank you for the suggestion. If alternative
> 1
> > > > turns
> > > > > > out
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > appropriate later, I will improve this part
> of
> > the
> > > > > > > content.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As the above discussion moves forward, the
> > > > option 2
> > > > > > > > > solution
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > Yes, after discussing with Shammon offline,
> we
> > > > think
> > > > > > that
> > > > > > > > > > > solution
> > > > > > > > > > > > 2
> > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > be more suitable and also avoid any
> > inconsistency
> > > > > > issues.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are some inconsistent descriptions in
> > the
> > > > > > > content.
> > > > > > > > > > Would
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > I will do my best to improve the document and
> > > > > > appreciate
> > > > > > > > your
> > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > can you put the unselected option in
> > `Rejected
> > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > Alternatives`.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I found there are two options in `Proposed
> > > > > Changes`,
> > > > > > > can
> > > > > > > > > you
> > > > > > > > > > > put
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > unselected option in `Rejected
> > Alternatives`? I
> > > > > think
> > > > > > > > this
> > > > > > > > > > may
> > > > > > > > > > > > help
> > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am still
> > > > confused
> > > > > > > after
> > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog()
> and
> > > > > > > > addCatalog()
> > > > > > > > > > have
> > > > > > > > > > > > no
> > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > difference based on their names. One of
> > them is
> > > > > > > > > responsible
> > > > > > > > > > > for
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> > Catalog>
> > > > > > > catalogs
> > > > > > > > is
> > > > > > > > > > > used
> > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > persistence.
> > > > I
> > > > > > > would
> > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > their purpose conceptually and clearly in
> > the
> > > > > FLIP.
> > > > > > > > Some
> > > > > > > > > > > common
> > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > features should be implemented, i.e. data
> > in
> > > > the
> > > > > > > cache
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> > should be
> > > > > > found
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > store
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the cache(either it has been initialized
> > or it
> > > > > will
> > > > > > > be
> > > > > > > > > lazy
> > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > for the same catalog name. The
> consistency
> > will
> > > > > be
> > > > > > > > taken
> > > > > > > > > > care
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > 3. As the above discussion moves forward,
> > the
> > > > > > option
> > > > > > > 2
> > > > > > > > > > > solution
> > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > like a replacement of option 1, because,
> > afaiu,
> > > > > > > issues
> > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > previously with option 1 are not solved
> > yet. Do
> > > > > you
> > > > > > > > still
> > > > > > > > > > > want
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > both options and ask for suggestions for
> > both
> > > > of
> > > > > > > them?
> > > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there are
> > some
> > > > > > > > > inconsistent
> > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the content. Would you like to clean
> them
> > up?
> > > > > > > Thanks!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thank you for your response, and I
> > completely
> > > > > > agree
> > > > > > > > > with
> > > > > > > > > > > your
> > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > Initially, I may have over complicated
> > the
> > > > > whole
> > > > > > > > issue.
> > > > > > > > > > > First
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > we need to consider the persistence of
> > the
> > > > > > > Catalog's
> > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > If we only need to provide persistence
> > for
> > > > > > Catalog
> > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> > Catalog
> > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > instance
> > > > to a
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > properties, and the default
> > implementation
> > > > will
> > > > > > > throw
> > > > > > > > > an
> > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > properties
> > > > of
> > > > > > the
> > > > > > > > > > catalog
> > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > * @return a map containing the
> > properties
> > > > of
> > > > > > the
> > > > > > > > > > catalog
> > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > * @throws
> > UnsupportedOperationException if
> > > > > the
> > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > * the default implementation of this
> > > > method
> > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > default Map<String, String>
> > toProperties()
> > > > {
> > > > > > > > > > > > > > > > > > > throw new
> > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > CatalogStore,
> > > > > > the
> > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > will be called when registering a
> Catalog
> > > > > > instance
> > > > > > > > with
> > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > Catalog
> > > > > > > catalog).
> > > > > > > > > The
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> > String>
> > > > > > > properties
> > > > > > > > > and
> > > > > > > > > > > > saved
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog instances
> > do
> > > > not
> > > > > > > > > implement
> > > > > > > > > > > this
> > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > CatalogStore,
> > > > > the
> > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > will not be called, ensuring
> consistency
> > with
> > > > > the
> > > > > > > > > > original
> > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> > > > > catalogs
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > we can start from the Catalog itself
> and
> > > > > provide
> > > > > > a
> > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > implementation for lazy loading, such
> as
> > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > implements
> > > > > > > Catalog
> > > > > > > > {
> > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> String>
> > > > > > > properties)
> > > > > > > > {
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon
> > FY <
> > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I have no questions about the
> > functionality
> > > > > of
> > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > > `registerCatalog`
> > > > > > > > or
> > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > `CatalogManager`. The implementation
> in
> > > > Trino
> > > > > > is
> > > > > > > > also
> > > > > > > > > > the
> > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only has
> one
> > > > > > > > > `createCatalog`,
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > > > > > `registerCatalog`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> Catalog>
> > > > > > catalogs`
> > > > > > > > or
> > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> > before,
> > > > the
> > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > `CatalogManager`.
> > > > How
> > > > > > > about
> > > > > > > > > > > saving
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > > > > `CatalogStore
> > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng
> > Jin <
> > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am very
> > sorry
> > > > > for
> > > > > > > the
> > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > by my deviation from the original
> > > > > discussion.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion
> > [1]
> > > > for
> > > > > > > this
> > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > discussion
> > > > > > before.
> > > > > > > > > > > However,
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > whole solution, I found that the
> > logic of
> > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > need to change much. *We cannot
> only
> > > > > persist
> > > > > > > > > Catalog
> > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > so exposing only
> > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> > because
> > > > in
> > > > > > the
> > > > > > > > end
> > > > > > > > > we
> > > > > > > > > > > > still
> > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > the configurations corresponding to
> > the
> > > > > > Catalog
> > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > decided to introduce the
> CatalogStore
> > > > > > interface
> > > > > > > > for
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > persistence. Regarding this part, I
> > also
> > > > > took
> > > > > > > > > > > inspiration
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > catalog
> > > > with
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > name
> > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I
> will
> > > > > provide
> > > > > > a
> > > > > > > > > > detailed
> > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > code examples for this part, and
> add
> > it
> > > > to
> > > > > > the
> > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > `addCatalog`
> > > > > still
> > > > > > > > looks
> > > > > > > > > > > > confused
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > this method is for storing catalog
> to
> > > > > > > > CatalogStore,
> > > > > > > > > > how
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> personal
> > > > > > opinion,
> > > > > > > > you
> > > > > > > > > > can
> > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive
> to
> > me.
> > > > I
> > > > > > > don't
> > > > > > > > > see
> > > > > > > > > > > any
> > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement,
> > which
> > > > > > method
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`?
> > I'm
> > > > > > > wondering
> > > > > > > > > > > whether
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> > design
> > > > was
> > > > > to
> > > > > > > add
> > > > > > > > > it
> > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> disrupt
> > the
> > > > > > > existing
> > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > configuration to
> > > > > the
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > the Catalog instance at the same
> time
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> > provide a
> > > > > > > default
> > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > I think it is necessary, otherwise
> we
> > > > would
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > Map to store the configuration for
> > lazy
> > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> registration
> > for
> > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > I don't think
> registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > made into an asynchronous interface
> > > > because
> > > > > > > > Catalog
> > > > > > > > > > is
> > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > initialization
> > > > > for
> > > > > > > > > catalog
> > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > inaccurate. It
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > registration. I
> > > > > have
> > > > > > > > > already
> > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> > Shammon
> > > > FY <
> > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > `registerCatalog`
> > > > > > > method
> > > > > > > > in
> > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > confuse users whether a method
> > named
> > > > > > > > `addCatalog`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the memory
> > > > catalog
> > > > > > is a
> > > > > > > > > > > `cache`,
> > > > > > > > > > > > I
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > of `cache` should not be exposed
> to
> > > > > users.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion
> > [1]
> > > > for
> > > > > > > this
> > > > > > > > > > FLIP.
> > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion
> of
> > that
> > > > > > > > > discussion
> > > > > > > > > > is
> > > > > > > > > > > > to
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an interface
> > and
> > > > > > > implement
> > > > > > > > it
> > > > > > > > > > for
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > as memory, file and external
> > system.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think there is a gap between
> the
> > > > > current
> > > > > > > FLIP
> > > > > > > > > > > design
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > What about the proposal of the
> > > > discussion
> > > > > > in
> > > > > > > > > thread
> > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM
> > yuxia <
> > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this
> > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > catalog
> > > > > with
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > name
> > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > complex to me. I think it'll be
> > > > better
> > > > > to
> > > > > > > > > explain
> > > > > > > > > > > it
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > these methods and give a brief
> > > > example
> > > > > in
> > > > > > > > this
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > `addCatalog`
> > > > > > still
> > > > > > > > > looks
> > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > catalog to
> > > > > > > > > > CatalogStore,
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > personal
> > > > > > > > opinion,
> > > > > > > > > > you
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> statement,
> > > > which
> > > > > > > method
> > > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > `addCatalog`?
> > > > I'm
> > > > > > > > > wondering
> > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > stement.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> > > > provide a
> > > > > > > > default
> > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > interface
> `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > registration
> > > > for
> > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog with
> > CREATE
> > > > > > > > CATALOG,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration method is used by
> > > > default.
> > > > > > > > > > > > > > > > > > > > > > > If asynchronous registration is
> > the
> > > > > > default
> > > > > > > > > > > behavior,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > synchronous
> > > > > > > > > registration
> > > > > > > > > > > just
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > Will both method `addCatalog`
> and
> > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > registration,
> > > > it
> > > > > > may
> > > > > > > > well
> > > > > > > > > > > that
> > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > executes successfully, but then
> > the
> > > > > > > following
> > > > > > > > > > > CREATE
> > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail to
> > open.
> > > > > > > > > > > > > > > > > > > > > > > I think it's a break change
> which
> > > > > should
> > > > > > be
> > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > > registration, I
> > > > > > > > > would
> > > > > > > > > > > > like
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > will be an executor to open or
> > > > register
> > > > > > > > catalog
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > > > > initialized
> > > > > > > if
> > > > > > > > it
> > > > > > > > > > has
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> > process
> > > > > fails,
> > > > > > > > these
> > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > initialization
> > > > > > for
> > > > > > > > > > catalog
> > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > jinfeng1995@gmail.com
> > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> dev@flink.apache.org
> > >
> > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> > Support
> > > > > > > > persistence
> > > > > > > > > > of
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for
> > your
> > > > > > > interest
> > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog with
> > the
> > > > same
> > > > > > > name
> > > > > > > > > that
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> > point.
> > > > > Based
> > > > > > on
> > > > > > > > my
> > > > > > > > > > > > current
> > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > understanding, the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > > will
> > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs and
> > have
> > > > the
> > > > > > > > highest
> > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There are three methods that
> can
> > > > affect
> > > > > > the
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This method puts the catalog
> > instance
> > > > > > into
> > > > > > > > the
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This method removes the catalog
> > > > > instance
> > > > > > > > > > > > corresponding
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> catalogName)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > > > > > corresponding
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > catalogs. If
> > > > > the
> > > > > > > > > catalog
> > > > > > > > > > > > does
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > > configuration
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > > > > initialized
> > > > > > > > > Catalog
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The following two methods only
> > modify
> > > > > the
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> catalogName,
> > > > > > > Map<String,
> > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This method saves the
> properties
> > to
> > > > the
> > > > > > > > > > > catalogStore
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > whether there is a catalogName
> > with
> > > > the
> > > > > > > same
> > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > This method removes the
> specified
> > > > > > > > configuration
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> catalogStore.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The following are possible
> > conflict
> > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > catalogName
> > > > > > > already
> > > > > > > > > > > exists
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > > Map<String,
> > > > > > > > > Catalog>,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > > catalog)
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > succeed and be directly saved
> to
> > the
> > > > > > > > > Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > catalogName
> > > > > > > already
> > > > > > > > > > > exists
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> Map<String,
> > > > > > Catalog>,
> > > > > > > > the
> > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog)
> > method
> > > > > will
> > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > catalogName
> > > > > > > already
> > > > > > > > > > > exists
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > addCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > String> properties) method can
> > > > directly
> > > > > > > save
> > > > > > > > > the
> > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > getCatalog(String
> > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > > initialization
> > > > > > > > > because
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > catalog instance already exists
> > in
> > > > > > catalogs
> > > > > > > > and
> > > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > corresponding to
> > > > > the
> > > > > > > > > original
> > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse users
> > that
> > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)`
> in
> > the
> > > > > > > > > `Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > catalogName,
> > > > > > > > > Map<String,
> > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This could potentially lead to
> > > > > > confusion. I
> > > > > > > > > > suggest
> > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > addCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > previously
> > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> > > > instance,
> > > > > > > > adding
> > > > > > > > > > the
> > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> > result in
> > > > > lazy
> > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > > encryption
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> necessary,
> > but
> > > > > > > perhaps
> > > > > > > > > the
> > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > should be implemented in the
> > > > > CatalogStore
> > > > > > > > when
> > > > > > > > > > the
> > > > > > > > > > > > data
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > FileCatalogStore
> > > > > > > > could
> > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > it is saved to a file, while
> the
> > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > does
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think it is necessary to
> have a
> > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > as
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> > Catalog
> > > > > > > > > > configurations
> > > > > > > > > > > in
> > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > implement
> > > > > > > > asynchronous
> > > > > > > > > > > > loading
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > additional
> > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > registration
> > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> concern.
> > > > When
> > > > > > > > > executing
> > > > > > > > > > > DDL,
> > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > statements that reference a
> > specific
> > > > > > > Catalog,
> > > > > > > > > the
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > will be initialized if it has
> not
> > > > been
> > > > > > > > > > initialized
> > > > > > > > > > > > yet.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > initialization process fails,
> > these
> > > > > > > > statements
> > > > > > > > > > will
> > > > > > > > > > > > not
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog>
> > > > catalogs
> > > > > > and
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have the
> > same
> > > > > > > catalog
> > > > > > > > > > name,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> > > > referring
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> Catalog
> > > > that
> > > > > > has
> > > > > > > > been
> > > > > > > > > > > most
> > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > will be given priority for
> > subsequent
> > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM
> > > > Xianxun
> > > > > > Ye
> > > > > > > <
> > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip,
> > I do
> > > > > > > believe
> > > > > > > > > this
> > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > for users. The firm I work for
> > also
> > > > > > > manages a
> > > > > > > > > lot
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes
> slow
> > > > > because
> > > > > > of
> > > > > > > > > > > loading a
> > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the
> > user's
> > > > > > Flink
> > > > > > > > SQL
> > > > > > > > > > > > through
> > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs
> to
> > > > > improve
> > > > > > > the
> > > > > > > > > > speed
> > > > > > > > > > > of
> > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I have
> > some
> > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog
> with
> > > > > CREATE
> > > > > > > > > CATALOG,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration method is used by
> > > > default.
> > > > > > > What
> > > > > > > > if
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > > > > asynchronous
> > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > question
> > > > with
> > > > > > > > > Shammon’s
> > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs
> and
> > > > > > > > CatalogStore
> > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the
> same
> > > > > catalog
> > > > > > > > name,
> > > > > > > > > > but
> > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> For
> > > > > example:
> > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > hiveCatalog
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> > > > > `catalogs`
> > > > > > > and
> > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan
> <
> > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Thanks Jingsong.
> Just naming, maybe `createCatalog` in TableEnv
+1 For this, I have already updated FLIP.
Best,
Feng
On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com> wrote:
> Thanks Feng,
>
> Just naming, maybe `createCatalog` in TableEnv, I can see many
> functions are converted to createxxx from registerxxx.
>
> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com> wrote:
> >
> > Hi jark, thanks for your suggestion.
> >
> > > 1. How to register the CatalogStore for Table API? I think the
> > CatalogStore should be immutable once TableEnv is created. Otherwise,
> there
> > might be some data inconsistencies when CatalogStore is changed.
> >
> > Yes, We should initialize the CatalogStore when creating the TableEnv.
> > Therefore, my current proposal is to add a method to configure the
> > CatalogStore in EnvironmentSettings.
> >
> > // Initialize a catalog Store
> > CatalogStore catalogStore = new FileCatalogStore("");
> >
> > // set up the Table API
> > final EnvironmentSettings settings =
> > EnvironmentSettings.newInstance().inBatchMode()
> > .withCatalogStore(catalogStore)
> > .build();
> >
> > final TableEnvironment tableEnv =
> TableEnvironment.create(settings);
> >
> >
> > > 2. Why does the CatalogStoreFactory interface only have a default
> method,
> > not an interface method?
> >
> > Sorry, While I did refer to the Catalog interface, I agree that as a new
> > interface, the CatalogStoreFactory should not have a default method but
> an
> > interface method. I have already modified the interface.
> >
> > > 3. Please mention the alternative API in Javadoc for the
> > deprecated`registerCatalog`.
> > > 4. In the "Compatibility" section, would be better to mention the
> changed
> > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > persisted catalog store) is used.
> >
> > Thanks for the suggestion, I have updated the FLIP [1].
> >
> >
> > [1].
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >
> >
> > Best,
> > Feng
> >
> > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> >
> > > Hi Feng,
> > >
> > > This is a useful FLIP. Thanks for starting this discussion.
> > > The current design looks pretty good to me. I just have some minor
> > > comments.
> > >
> > > 1. How to register the CatalogStore for Table API? I think the
> CatalogStore
> > > should be immutable once TableEnv is created. Otherwise, there might be
> > > some data inconsistencies when CatalogStore is changed.
> > >
> > > 2. Why does the CatalogStoreFactory interface only have a default
> method,
> > > not an interface method?
> > >
> > > 3. Please mention the alternative API in Javadoc for the deprecated
> > > `registerCatalog`.
> > >
> > > 4. In the "Compatibility" section, would be better to mention the
> changed
> > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > persisted catalog store) is used.
> > >
> > >
> > > Best,
> > > Jark
> > >
> > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
> > >
> > > > Hi ， thanks all for reviewing the flip.
> > > >
> > > > @Ron
> > > >
> > > > > Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > > need
> > > > to provide a default implementation?
> > > >
> > > > Yes, we will provide a default InMemoryCatalogStoreFactory to create
> an
> > > > InMemoryCatalogStore.
> > > >
> > > > > If we get a Catalog from CatalogStore, after initializing it,
> whether
> > > we
> > > > put it in Map<String, Catalog> catalogs again?
> > > >
> > > > Yes, in the current design, catalogs are stored as snapshots, and
> once
> > > > initialized, the Catalog will be placed in the Map<String, Catalog>
> > > > catalogs.
> > > > Subsequently, the Map<String, Catalog> catalogs will be the primary
> > > source
> > > > for obtaining the corresponding Catalog.
> > > >
> > > > > how about renaming them to `catalog.store.type` and
> > > > `catalog.store.path`?
> > > >
> > > > I think it is okay. Adding "sql" at the beginning may seem a bit
> > > strange. I
> > > > will update the FLIP.
> > > >
> > > >
> > > >
> > > > @Shammon
> > > >
> > > > Thank you for the review. I have made the necessary corrections.
> > > > Regarding the modifications made to the Public Interface, I have also
> > > > included the relevant changes to the `TableEnvironment`.
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > >
> > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com>
> wrote:
> > > >
> > > > > Hi feng,
> > > > >
> > > > > Thanks for updating, I have some minor comments
> > > > >
> > > > > 1. The modification of `CatalogManager` should not be in `Public
> > > > > Interfaces`, it is not a public interface.
> > > > >
> > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > `CatalogStoreFactory`
> > > > >
> > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > catalogStore.get(catalogName);`
> > > > >
> > > > > Best,
> > > > > Shammon FY
> > > > >
> > > > >
> > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi, Feng
> > > > > >
> > > > > > Thanks for driving this FLIP, this proposal is very useful for
> > > catalog
> > > > > > management.
> > > > > > I have some small questions:
> > > > > >
> > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore method,
> do we
> > > > > need
> > > > > > to provide a default implementation?
> > > > > > 2. If we get Catalog from CatalogStore, after initializing it,
> > > whether
> > > > we
> > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > >
> > > > > > Best,
> > > > > > Ron
> > > > > >
> > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > >
> > > > > > > Hi yuxia
> > > > > > >
> > > > > > > > But from the code in Proposed Changes, once we register the
> > > > Catalog,
> > > > > > we
> > > > > > > initialize it and open it. right?
> > > > > > >
> > > > > > > Yes, In order to avoid inconsistent semantics of the original
> > > CREATE
> > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > registerCatalog
> > > > so
> > > > > > > that parameter validation can be performed.
> > > > > > >
> > > > > > > In the current design, lazy initialization is mainly reflected
> in
> > > > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > > > configurations,
> > > > > > > only initialization is required in getCatalog.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> luoyuxia@alumni.sjtu.edu.cn
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Feng.
> > > > > > > > I'm trying to understanding the meaning of *lazy
> initialization*.
> > > > If
> > > > > > i'm
> > > > > > > > wrong, please correct me.
> > > > > > > >
> > > > > > > > IIUC, lazy initialization means only you need to access the
> > > > catalog,
> > > > > > then
> > > > > > > > you initialize it. But from the code in Proposed Changes,
> once we
> > > > > > > register
> > > > > > > > the Catalog,
> > > > > > > > we initialize it and open it. right?
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Yuxia
> > > > > > > >
> > > > > > > > ----- 原始邮件 -----
> > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > > configuration
> > > > > > > > and asynchronous registration
> > > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > >
> > > > > > > > One of the major changes is that current design will provide
> > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> cache,
> > > > which
> > > > > > > means
> > > > > > > > once it has been initialized, any changes done by other
> sessions
> > > > will
> > > > > > not
> > > > > > > > affect it. Point 6 described follow-up options for further
> > > > > improvement.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> jinfeng1995@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all, I would like to update you on the latest progress
> of
> > > the
> > > > > > FLIP.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> ShengKai
> > > > Fang
> > > > > > > and I
> > > > > > > > > had an offline discussion regarding the overall solution
> for
> > > > Flink
> > > > > > > > > CatalogStore. We have reached a consensus and I have
> updated
> > > the
> > > > > > final
> > > > > > > > > solution in FLIP.
> > > > > > > > >
> > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > >
> > > > > > > > > Introduce CatalogDescriptor to store catalog
> configuration
> > > > > similar
> > > > > > > to
> > > > > > > > > TableDescriptor.
> > > > > > > > > 2.
> > > > > > > > >
> > > > > > > > > The two key functions of CatalogStore - void
> > > > storeCatalog(String
> > > > > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > > > > getCatalog(String)
> > > > > > > > > will both use CatalogDescriptor instead of Catalog
> instance.
> > > > > This
> > > > > > > way,
> > > > > > > > > CatalogStore will only be responsible for saving and
> > > > retrieving
> > > > > > > > catalog
> > > > > > > > > configurations without having to initialize catalogs.
> > > > > > > > > 3.
> > > > > > > > >
> > > > > > > > > The default registerCatalog(String catalogName, Catalog
> > > > catalog)
> > > > > > > > > function in CatalogManager will be marked as deprecated.
> > > > > > > > > 4.
> > > > > > > > >
> > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > CatalogDescriptor
> > > > > > > > > catalog) will be added to serve as the default
> registration
> > > > > > function
> > > > > > > > for
> > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > 5.
> > > > > > > > >
> > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> remain
> > > > > > unchanged
> > > > > > > > and
> > > > > > > > > save initialized catalogs.This means that deletion
> > > operations
> > > > > from
> > > > > > > one
> > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > 6.
> > > > > > > > >
> > > > > > > > > To support multi-session synchronization scenarios for
> > > > deletions
> > > > > > > later
> > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > configurable.There
> > > > > > may
> > > > > > > > be
> > > > > > > > > three possible situations:
> > > > > > > > >
> > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > >
> > > > > > > > > b.Introduction of LRU cache logic which can
> automatically
> > > > clear
> > > > > > > > > long-unused catalogs.
> > > > > > > > >
> > > > > > > > > c.No caching of any instances; each call to getCatalog
> > > > creates a
> > > > > > new
> > > > > > > > > instance.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is the document for discussion:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > >
> > > > > > > > > This is the final proposal document:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thank you very much for your attention and suggestions on
> this
> > > > > > FLIP. A
> > > > > > > > > special thanks to Hang Ruan for his suggestions on the
> entire
> > > > > design
> > > > > > > and
> > > > > > > > > organizing offline discussions.
> > > > > > > > >
> > > > > > > > > If you have any further suggestions or feedback about this
> FLIP
> > > > > > please
> > > > > > > > feel
> > > > > > > > > free to share.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for improving the FLIP. It looks good to me. We
> could
> > > > > still
> > > > > > > > > > reconsider in the future how to provide more common
> built-in
> > > > > cache
> > > > > > > > > > functionality in CatalogManager so that not every
> > > CatalogSotre
> > > > > > > > > > implementation has to take care of it.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > jinfeng1995@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jing,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > >
> > > > > > > > > > > > There might be more such issues. I would suggest you
> > > > > > completely
> > > > > > > > walk
> > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > >
> > > > > > > > > > > I am very sorry for my carelessness and at the same
> time, I
> > > > > > greatly
> > > > > > > > > > > appreciate your careful review. I have thoroughly
> checked
> > > the
> > > > > > > entire
> > > > > > > > > FLIP
> > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > > > > > CatalogManager
> > > > > > > > > > > could work without Optional CatalogStore being
> configured.
> > > > > > > > > > >
> > > > > > > > > > > Yes, in the original design, CatalogStore was not
> necessary
> > > > > > because
> > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs to
> store
> > > > > > catalog
> > > > > > > > > > > instances.
> > > > > > > > > > > However, this caused inconsistency issues. Therefore, I
> > > > > modified
> > > > > > > this
> > > > > > > > > > part
> > > > > > > > > > > of the design and removed Map<String, Catalog> catalogs
> > > from
> > > > > > > > > > > CatalogManager.
> > > > > > > > > > > At the same time, InMemoryCatalog will serve as the
> > > default
> > > > > > > > > CatalogStore
> > > > > > > > > > > to save catalogs in memory and replace the
> functionality of
> > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > The previous plan that kept
> Map<String,Catalog>catalogs has
> > > > > been
> > > > > > > > moved
> > > > > > > > > to
> > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > There are still many places contain inconsistent
> content,
> > > > > e.g.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > 2. The java comment of the method
> registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog catalog) in CatalogManager does not tell
> what the
> > > > > > method
> > > > > > > is
> > > > > > > > > > > doing.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > There might be more such issues. I would suggest you
> > > > > completely
> > > > > > > > walk
> > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > >
> > > > > > > > > > > > About the inMemoryCatalogStore, do you mean that you
> will
> > > > > build
> > > > > > > the
> > > > > > > > > > cache
> > > > > > > > > > > > functionality in the CatalogStore? This is a very
> > > different
> > > > > > > design
> > > > > > > > > > > concept
> > > > > > > > > > > > from what the current FLIP described. If I am not
> > > mistaken,
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > current FLIP design, CatalogManager could work
> without
> > > > > Optional
> > > > > > > > > > > > CatalogStore being configured. That is the reason
> why I
> > > > > > mentioned
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > last email that the example code wrt the Optional
> > > > > CatalogStore
> > > > > > is
> > > > > > > > not
> > > > > > > > > > > > correct.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you very much for the careful review. I have
> > > > already
> > > > > > made
> > > > > > > > the
> > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > multi-instance
> > > > > > > > > scenarios, I
> > > > > > > > > > > am
> > > > > > > > > > > > > not
> > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > >
> > > > > > > > > > > > > Currently, there are conflicts in multiple
> scenarios
> > > with
> > > > > the
> > > > > > > > > current
> > > > > > > > > > > > > design. I am thinking whether we should remove
> > > > 'Map<String,
> > > > > > > > > Catalog>'
> > > > > > > > > > > and
> > > > > > > > > > > > > make Cache the default behavior of
> > > InMemoryCatalogStore.
> > > > > This
> > > > > > > > way,
> > > > > > > > > > > users
> > > > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > > > multi-instance
> > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for working on the FLIP. There are still
> some
> > > > NIT
> > > > > > > issues
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > like:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been
> used
> > > as
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > instead of Optional in the code example. It
> should be
> > > > > fine
> > > > > > to
> > > > > > > > use
> > > > > > > > > > it
> > > > > > > > > > > as
> > > > > > > > > > > > > > pseudo code for now and update it after you
> submit
> > > the
> > > > > PR.
> > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> > > > > rejected
> > > > > > > > > section
> > > > > > > > > > > > which
> > > > > > > > > > > > > > should be persistContext(...) to keep it
> consistent.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Speaking of the conflict issues in the
> multi-instance
> > > > > > > > scenarios,
> > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > > > not
> > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > used as a cache, it should be invalid, once the
> > > related
> > > > > > > catalog
> > > > > > > > > has
> > > > > > > > > > > > been
> > > > > > > > > > > > > > removed from the CatalogStore by another
> instance.
> > > Did
> > > > I
> > > > > > miss
> > > > > > > > > > > > something?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > I think this is a good function name, I have
> > > updated
> > > > it
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Some common cache features should be
> > > implemented
> > > > > > > > > > > > > > > Thank you for the suggestion. If alternative 1
> > > turns
> > > > > out
> > > > > > to
> > > > > > > > be
> > > > > > > > > > more
> > > > > > > > > > > > > > > appropriate later, I will improve this part of
> the
> > > > > > content.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As the above discussion moves forward, the
> > > option 2
> > > > > > > > solution
> > > > > > > > > > > looks
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > Yes, after discussing with Shammon offline, we
> > > think
> > > > > that
> > > > > > > > > > solution
> > > > > > > > > > > 2
> > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > be more suitable and also avoid any
> inconsistency
> > > > > issues.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are some inconsistent descriptions in
> the
> > > > > > content.
> > > > > > > > > Would
> > > > > > > > > > > you
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > I will do my best to improve the document and
> > > > > appreciate
> > > > > > > your
> > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > can you put the unselected option in
> `Rejected
> > > > > > > > Alternatives`
> > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > Alternatives`.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I found there are two options in `Proposed
> > > > Changes`,
> > > > > > can
> > > > > > > > you
> > > > > > > > > > put
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > unselected option in `Rejected
> Alternatives`? I
> > > > think
> > > > > > > this
> > > > > > > > > may
> > > > > > > > > > > help
> > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am still
> > > confused
> > > > > > after
> > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > > > > > addCatalog()
> > > > > > > > > have
> > > > > > > > > > > no
> > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > difference based on their names. One of
> them is
> > > > > > > > responsible
> > > > > > > > > > for
> > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> Catalog>
> > > > > > catalogs
> > > > > > > is
> > > > > > > > > > used
> > > > > > > > > > > > as a
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > and catalogStore is used for data
> persistence.
> > > I
> > > > > > would
> > > > > > > > > > suggest
> > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > their purpose conceptually and clearly in
> the
> > > > FLIP.
> > > > > > > Some
> > > > > > > > > > common
> > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > features should be implemented, i.e. data
> in
> > > the
> > > > > > cache
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> should be
> > > > > found
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > store
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the cache(either it has been initialized
> or it
> > > > will
> > > > > > be
> > > > > > > > lazy
> > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > for the same catalog name. The consistency
> will
> > > > be
> > > > > > > taken
> > > > > > > > > care
> > > > > > > > > > > of
> > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > 3. As the above discussion moves forward,
> the
> > > > > option
> > > > > > 2
> > > > > > > > > > solution
> > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > like a replacement of option 1, because,
> afaiu,
> > > > > > issues
> > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > previously with option 1 are not solved
> yet. Do
> > > > you
> > > > > > > still
> > > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > both options and ask for suggestions for
> both
> > > of
> > > > > > them?
> > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there are
> some
> > > > > > > > inconsistent
> > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the content. Would you like to clean them
> up?
> > > > > > Thanks!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you for your response, and I
> completely
> > > > > agree
> > > > > > > > with
> > > > > > > > > > your
> > > > > > > > > > > > > point
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > Initially, I may have over complicated
> the
> > > > whole
> > > > > > > issue.
> > > > > > > > > > First
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > we need to consider the persistence of
> the
> > > > > > Catalog's
> > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > If we only need to provide persistence
> for
> > > > > Catalog
> > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> Catalog
> > > > > > > interface.
> > > > > > > > > > > > > > > > > > This method can convert a Catalog
> instance
> > > to a
> > > > > > > > > Map<String,
> > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > properties, and the default
> implementation
> > > will
> > > > > > throw
> > > > > > > > an
> > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > * Returns a map containing the
> properties
> > > of
> > > > > the
> > > > > > > > > catalog
> > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > * @return a map containing the
> properties
> > > of
> > > > > the
> > > > > > > > > catalog
> > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > * @throws
> UnsupportedOperationException if
> > > > the
> > > > > > > > > > > implementing
> > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > * the default implementation of this
> > > method
> > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > default Map<String, String>
> toProperties()
> > > {
> > > > > > > > > > > > > > > > > > throw new
> > > > > UnsupportedOperationException("Please
> > > > > > > > > > implement
> > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > CatalogStore,
> > > > > the
> > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > will be called when registering a Catalog
> > > > > instance
> > > > > > > with
> > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> Catalog
> > > > > > catalog).
> > > > > > > > The
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> String>
> > > > > > properties
> > > > > > > > and
> > > > > > > > > > > saved
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog instances
> do
> > > not
> > > > > > > > implement
> > > > > > > > > > this
> > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. If the user does not use a
> CatalogStore,
> > > > the
> > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > will not be called, ensuring consistency
> with
> > > > the
> > > > > > > > > original
> > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> > > > catalogs
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > we can start from the Catalog itself and
> > > > provide
> > > > > a
> > > > > > > > > > dedicated
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> implements
> > > > > > Catalog
> > > > > > > {
> > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > > > > > properties)
> > > > > > > {
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon
> FY <
> > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have no questions about the
> functionality
> > > > of
> > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > `registerCatalog`
> > > > > > > or
> > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > `CatalogManager`. The implementation in
> > > Trino
> > > > > is
> > > > > > > also
> > > > > > > > > the
> > > > > > > > > > > > same,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > > > > > `createCatalog`,
> > > > > > > > > > > which
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > > > > `registerCatalog`
> > > > > > > > > or
> > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> > > > > catalogs`
> > > > > > > or
> > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> before,
> > > the
> > > > > > > > > `Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> `CatalogManager`.
> > > How
> > > > > > about
> > > > > > > > > > saving
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > > > `CatalogStore
> > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng
> Jin <
> > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am very
> sorry
> > > > for
> > > > > > the
> > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > by my deviation from the original
> > > > discussion.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion
> [1]
> > > for
> > > > > > this
> > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> discussion
> > > > > before.
> > > > > > > > > > However,
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > whole solution, I found that the
> logic of
> > > > > > > > > > CatalogManager
> > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > need to change much. *We cannot only
> > > > persist
> > > > > > > > Catalog
> > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > so exposing only
> registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> because
> > > in
> > > > > the
> > > > > > > end
> > > > > > > > we
> > > > > > > > > > > still
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > the configurations corresponding to
> the
> > > > > Catalog
> > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> > > > > interface
> > > > > > > for
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > persistence. Regarding this part, I
> also
> > > > took
> > > > > > > > > > inspiration
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> catalog
> > > with
> > > > > the
> > > > > > > > same
> > > > > > > > > > name
> > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I will
> > > > provide
> > > > > a
> > > > > > > > > detailed
> > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > code examples for this part, and add
> it
> > > to
> > > > > the
> > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> `addCatalog`
> > > > still
> > > > > > > looks
> > > > > > > > > > > confused
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > > CatalogStore,
> > > > > > > > > how
> > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > > opinion,
> > > > > > > you
> > > > > > > > > can
> > > > > > > > > > > > decide
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to
> me.
> > > I
> > > > > > don't
> > > > > > > > see
> > > > > > > > > > any
> > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement,
> which
> > > > > method
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`?
> I'm
> > > > > > wondering
> > > > > > > > > > whether
> > > > > > > > > > > > user
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> design
> > > was
> > > > to
> > > > > > add
> > > > > > > > it
> > > > > > > > > > > > directly
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore, but this would disrupt
> the
> > > > > > existing
> > > > > > > > > > logic.
> > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > we can do both: save the
> configuration to
> > > > the
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> provide a
> > > > > > default
> > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > I think it is necessary, otherwise we
> > > would
> > > > > > need
> > > > > > > to
> > > > > > > > > > > > introduce
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > Map to store the configuration for
> lazy
> > > > > > loading.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 4: About asynchronous registration
> for
> > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > made into an asynchronous interface
> > > because
> > > > > > > Catalog
> > > > > > > > > is
> > > > > > > > > > > > > already
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> initialization
> > > > for
> > > > > > > > catalog
> > > > > > > > > > than
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > Indeed, my description was
> inaccurate. It
> > > > > > should
> > > > > > > be
> > > > > > > > > > lazy
> > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > instead of asynchronous
> registration. I
> > > > have
> > > > > > > > already
> > > > > > > > > > > > updated
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> Shammon
> > > FY <
> > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think if there is a
> `registerCatalog`
> > > > > > method
> > > > > > > in
> > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > confuse users whether a method
> named
> > > > > > > `addCatalog`
> > > > > > > > > or
> > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > And as you mentioned, the memory
> > > catalog
> > > > > is a
> > > > > > > > > > `cache`,
> > > > > > > > > > > I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > of `cache` should not be exposed to
> > > > users.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion
> [1]
> > > for
> > > > > > this
> > > > > > > > > FLIP.
> > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of
> that
> > > > > > > > discussion
> > > > > > > > > is
> > > > > > > > > > > to
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an interface
> and
> > > > > > implement
> > > > > > > it
> > > > > > > > > for
> > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > as memory, file and external
> system.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think there is a gap between the
> > > > current
> > > > > > FLIP
> > > > > > > > > > design
> > > > > > > > > > > > and
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > What about the proposal of the
> > > discussion
> > > > > in
> > > > > > > > thread
> > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM
> yuxia <
> > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this
> FLIP.
> > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> catalog
> > > > with
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > name
> > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > complex to me. I think it'll be
> > > better
> > > > to
> > > > > > > > explain
> > > > > > > > > > it
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > these methods and give a brief
> > > example
> > > > in
> > > > > > > this
> > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> `addCatalog`
> > > > > still
> > > > > > > > looks
> > > > > > > > > > > > confused
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > this method is for storing
> catalog to
> > > > > > > > > CatalogStore,
> > > > > > > > > > > how
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> personal
> > > > > > > opinion,
> > > > > > > > > you
> > > > > > > > > > > can
> > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement,
> > > which
> > > > > > method
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> `addCatalog`?
> > > I'm
> > > > > > > > wondering
> > > > > > > > > > > > whether
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> stement.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> > > provide a
> > > > > > > default
> > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> registration
> > > for
> > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > When creating a catalog with
> CREATE
> > > > > > > CATALOG,
> > > > > > > > > the
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration method is used by
> > > default.
> > > > > > > > > > > > > > > > > > > > > > If asynchronous registration is
> the
> > > > > default
> > > > > > > > > > behavior,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > that user can switch to
> synchronous
> > > > > > > > registration
> > > > > > > > > > just
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > > > > > `registerCatalog`
> > > > > > > > > > > be
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> registration,
> > > it
> > > > > may
> > > > > > > well
> > > > > > > > > > that
> > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > executes successfully, but then
> the
> > > > > > following
> > > > > > > > > > CREATE
> > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail to
> open.
> > > > > > > > > > > > > > > > > > > > > > I think it's a break change which
> > > > should
> > > > > be
> > > > > > > > > > > highlighted
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > registration, I
> > > > > > > > would
> > > > > > > > > > > like
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > will be an executor to open or
> > > register
> > > > > > > catalog
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > > > initialized
> > > > > > if
> > > > > > > it
> > > > > > > > > has
> > > > > > > > > > > not
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> process
> > > > fails,
> > > > > > > these
> > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > initialization
> > > > > for
> > > > > > > > > catalog
> > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > jinfeng1995@gmail.com
> > > > >
> > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <dev@flink.apache.org
> >
> > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> 3:27:45
> > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> Support
> > > > > > > persistence
> > > > > > > > > of
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for
> your
> > > > > > interest
> > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog with
> the
> > > same
> > > > > > name
> > > > > > > > that
> > > > > > > > > > > > exists
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> point.
> > > > Based
> > > > > on
> > > > > > > my
> > > > > > > > > > > current
> > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > understanding, the Map<String,
> > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > > will
> > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs and
> have
> > > the
> > > > > > > highest
> > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There are three methods that can
> > > affect
> > > > > the
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > catalogName,
> > > > > > > Catalog
> > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This method puts the catalog
> instance
> > > > > into
> > > > > > > the
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This method removes the catalog
> > > > instance
> > > > > > > > > > > corresponding
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> > > catalogs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > > > > corresponding
> > > > > > > > > > catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> catalogs. If
> > > > the
> > > > > > > > catalog
> > > > > > > > > > > does
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > configuration
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > > > initialized
> > > > > > > > Catalog
> > > > > > > > > > > > instance
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The following two methods only
> modify
> > > > the
> > > > > > > > > > > configuration
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > > > > > Map<String,
> > > > > > > > > > String>
> > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This method saves the properties
> to
> > > the
> > > > > > > > > > catalogStore
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > whether there is a catalogName
> with
> > > the
> > > > > > same
> > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> catalogName)
> > > > > > > > > > > > > > > > > > > > > > This method removes the specified
> > > > > > > configuration
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The following are possible
> conflict
> > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> catalogName
> > > > > > already
> > > > > > > > > > exists
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > Map<String,
> > > > > > > > Catalog>,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > > catalog)
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > succeed and be directly saved to
> the
> > > > > > > > Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> catalogName
> > > > > > already
> > > > > > > > > > exists
> > > > > > > > > > > in
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> > > > > Catalog>,
> > > > > > > the
> > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog)
> method
> > > > will
> > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> catalogName
> > > > > > already
> > > > > > > > > > exists
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > addCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > String> properties) method can
> > > directly
> > > > > > save
> > > > > > > > the
> > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > getCatalog(String
> > > > > > > > > > catalogName)
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > initialization
> > > > > > > > because
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > catalog instance already exists
> in
> > > > > catalogs
> > > > > > > and
> > > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > unregisterCatalog(String
> > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > remove the instance
> corresponding to
> > > > the
> > > > > > > > original
> > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think it will confuse users
> that
> > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in
> the
> > > > > > > > `Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> catalogName,
> > > > > > > > Map<String,
> > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This could potentially lead to
> > > > > confusion. I
> > > > > > > > > suggest
> > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > addCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > String> properties), as
> previously
> > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> > > instance,
> > > > > > > adding
> > > > > > > > > the
> > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > registerCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> result in
> > > > lazy
> > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > encryption
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think encryption is necessary,
> but
> > > > > > perhaps
> > > > > > > > the
> > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > should be implemented in the
> > > > CatalogStore
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > > data
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > FileCatalogStore
> > > > > > > could
> > > > > > > > > > > encrypt
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > does
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > as
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> Catalog
> > > > > > > > > configurations
> > > > > > > > > > in
> > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> implement
> > > > > > > asynchronous
> > > > > > > > > > > loading
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > would need to introduce
> additional
> > > > cache.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> registration
> > > > > failed?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This is also a critical concern.
> > > When
> > > > > > > > executing
> > > > > > > > > > DDL,
> > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > statements that reference a
> specific
> > > > > > Catalog,
> > > > > > > > the
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > will be initialized if it has not
> > > been
> > > > > > > > > initialized
> > > > > > > > > > > yet.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > initialization process fails,
> these
> > > > > > > statements
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog>
> > > catalogs
> > > > > and
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have the
> same
> > > > > > catalog
> > > > > > > > > name,
> > > > > > > > > > > but
> > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> > > referring
> > > > > to
> > > > > > > the
> > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > key principle is that the Catalog
> > > that
> > > > > has
> > > > > > > been
> > > > > > > > > > most
> > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > will be given priority for
> subsequent
> > > > > use.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM
> > > Xianxun
> > > > > Ye
> > > > > > <
> > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip,
> I do
> > > > > > believe
> > > > > > > > this
> > > > > > > > > > > Flip
> > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > for users. The firm I work for
> also
> > > > > > manages a
> > > > > > > > lot
> > > > > > > > > > of
> > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes slow
> > > > because
> > > > > of
> > > > > > > > > > loading a
> > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the
> user's
> > > > > Flink
> > > > > > > SQL
> > > > > > > > > > > through
> > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs to
> > > > improve
> > > > > > the
> > > > > > > > > speed
> > > > > > > > > > of
> > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I have
> some
> > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog with
> > > > CREATE
> > > > > > > > CATALOG,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration method is used by
> > > default.
> > > > > > What
> > > > > > > if
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > > > asynchronous
> > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> question
> > > with
> > > > > > > > Shammon’s
> > > > > > > > > > > first
> > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > > > > > CatalogStore
> > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the same
> > > > catalog
> > > > > > > name,
> > > > > > > > > but
> > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > different catalog instances. For
> > > > example:
> > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> hiveCatalog
> > > > in
> > > > > > the
> > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> > > > `catalogs`
> > > > > > and
> > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Feng,
Just naming, maybe `createCatalog` in TableEnv, I can see many
functions are converted to createxxx from registerxxx.
On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com> wrote:
>
> Hi jark, thanks for your suggestion.
>
> > 1. How to register the CatalogStore for Table API? I think the
> CatalogStore should be immutable once TableEnv is created. Otherwise, there
> might be some data inconsistencies when CatalogStore is changed.
>
> Yes, We should initialize the CatalogStore when creating the TableEnv.
> Therefore, my current proposal is to add a method to configure the
> CatalogStore in EnvironmentSettings.
>
> // Initialize a catalog Store
> CatalogStore catalogStore = new FileCatalogStore("");
>
> // set up the Table API
> final EnvironmentSettings settings =
> EnvironmentSettings.newInstance().inBatchMode()
> .withCatalogStore(catalogStore)
> .build();
>
> final TableEnvironment tableEnv = TableEnvironment.create(settings);
>
>
> > 2. Why does the CatalogStoreFactory interface only have a default method,
> not an interface method?
>
> Sorry, While I did refer to the Catalog interface, I agree that as a new
> interface, the CatalogStoreFactory should not have a default method but an
> interface method. I have already modified the interface.
>
> > 3. Please mention the alternative API in Javadoc for the
> deprecated`registerCatalog`.
> > 4. In the "Compatibility" section, would be better to mention the changed
> behavior of CREATE CATALOG statement if FileCatalogStore (or other
> persisted catalog store) is used.
>
> Thanks for the suggestion, I have updated the FLIP [1].
>
>
> [1].
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>
>
> Best,
> Feng
>
> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
>
> > Hi Feng,
> >
> > This is a useful FLIP. Thanks for starting this discussion.
> > The current design looks pretty good to me. I just have some minor
> > comments.
> >
> > 1. How to register the CatalogStore for Table API? I think the CatalogStore
> > should be immutable once TableEnv is created. Otherwise, there might be
> > some data inconsistencies when CatalogStore is changed.
> >
> > 2. Why does the CatalogStoreFactory interface only have a default method,
> > not an interface method?
> >
> > 3. Please mention the alternative API in Javadoc for the deprecated
> > `registerCatalog`.
> >
> > 4. In the "Compatibility" section, would be better to mention the changed
> > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > persisted catalog store) is used.
> >
> >
> > Best,
> > Jark
> >
> > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
> >
> > > Hi ， thanks all for reviewing the flip.
> > >
> > > @Ron
> > >
> > > > Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > need
> > > to provide a default implementation?
> > >
> > > Yes, we will provide a default InMemoryCatalogStoreFactory to create an
> > > InMemoryCatalogStore.
> > >
> > > > If we get a Catalog from CatalogStore, after initializing it, whether
> > we
> > > put it in Map<String, Catalog> catalogs again?
> > >
> > > Yes, in the current design, catalogs are stored as snapshots, and once
> > > initialized, the Catalog will be placed in the Map<String, Catalog>
> > > catalogs.
> > > Subsequently, the Map<String, Catalog> catalogs will be the primary
> > source
> > > for obtaining the corresponding Catalog.
> > >
> > > > how about renaming them to `catalog.store.type` and
> > > `catalog.store.path`?
> > >
> > > I think it is okay. Adding "sql" at the beginning may seem a bit
> > strange. I
> > > will update the FLIP.
> > >
> > >
> > >
> > > @Shammon
> > >
> > > Thank you for the review. I have made the necessary corrections.
> > > Regarding the modifications made to the Public Interface, I have also
> > > included the relevant changes to the `TableEnvironment`.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com> wrote:
> > >
> > > > Hi feng,
> > > >
> > > > Thanks for updating, I have some minor comments
> > > >
> > > > 1. The modification of `CatalogManager` should not be in `Public
> > > > Interfaces`, it is not a public interface.
> > > >
> > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > `CatalogStoreFactory`
> > > >
> > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > catalogStore.get(catalogName);`
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > >
> > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
> > > >
> > > > > Hi, Feng
> > > > >
> > > > > Thanks for driving this FLIP, this proposal is very useful for
> > catalog
> > > > > management.
> > > > > I have some small questions:
> > > > >
> > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > > > need
> > > > > to provide a default implementation?
> > > > > 2. If we get Catalog from CatalogStore, after initializing it,
> > whether
> > > we
> > > > > put it to Map<String, Catalog> catalogs again?
> > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > `catalog.store.type` and `catalog.store.path`?
> > > > >
> > > > > Best,
> > > > > Ron
> > > > >
> > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > >
> > > > > > Hi yuxia
> > > > > >
> > > > > > > But from the code in Proposed Changes, once we register the
> > > Catalog,
> > > > > we
> > > > > > initialize it and open it. right?
> > > > > >
> > > > > > Yes, In order to avoid inconsistent semantics of the original
> > CREATE
> > > > > > CATALOG DDL, Catalog will be directly initialized in
> > registerCatalog
> > > so
> > > > > > that parameter validation can be performed.
> > > > > >
> > > > > > In the current design, lazy initialization is mainly reflected in
> > > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > > configurations,
> > > > > > only initialization is required in getCatalog.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <luoyuxia@alumni.sjtu.edu.cn
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Feng.
> > > > > > > I'm trying to understanding the meaning of *lazy initialization*.
> > > If
> > > > > i'm
> > > > > > > wrong, please correct me.
> > > > > > >
> > > > > > > IIUC, lazy initialization means only you need to access the
> > > catalog,
> > > > > then
> > > > > > > you initialize it. But from the code in Proposed Changes, once we
> > > > > > register
> > > > > > > the Catalog,
> > > > > > > we initialize it and open it. right?
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Yuxia
> > > > > > >
> > > > > > > ----- 原始邮件 -----
> > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > configuration
> > > > > > > and asynchronous registration
> > > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > >
> > > > > > > One of the major changes is that current design will provide
> > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a cache,
> > > which
> > > > > > means
> > > > > > > once it has been initialized, any changes done by other sessions
> > > will
> > > > > not
> > > > > > > affect it. Point 6 described follow-up options for further
> > > > improvement.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi all, I would like to update you on the latest progress of
> > the
> > > > > FLIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai
> > > Fang
> > > > > > and I
> > > > > > > > had an offline discussion regarding the overall solution for
> > > Flink
> > > > > > > > CatalogStore. We have reached a consensus and I have updated
> > the
> > > > > final
> > > > > > > > solution in FLIP.
> > > > > > > >
> > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > >
> > > > > > > > 1.
> > > > > > > >
> > > > > > > > Introduce CatalogDescriptor to store catalog configuration
> > > > similar
> > > > > > to
> > > > > > > > TableDescriptor.
> > > > > > > > 2.
> > > > > > > >
> > > > > > > > The two key functions of CatalogStore - void
> > > storeCatalog(String
> > > > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > > > getCatalog(String)
> > > > > > > > will both use CatalogDescriptor instead of Catalog instance.
> > > > This
> > > > > > way,
> > > > > > > > CatalogStore will only be responsible for saving and
> > > retrieving
> > > > > > > catalog
> > > > > > > > configurations without having to initialize catalogs.
> > > > > > > > 3.
> > > > > > > >
> > > > > > > > The default registerCatalog(String catalogName, Catalog
> > > catalog)
> > > > > > > > function in CatalogManager will be marked as deprecated.
> > > > > > > > 4.
> > > > > > > >
> > > > > > > > A new function registerCatalog(String catalogName,
> > > > > CatalogDescriptor
> > > > > > > > catalog) will be added to serve as the default registration
> > > > > function
> > > > > > > for
> > > > > > > > catalogs in CatalogManager.
> > > > > > > > 5.
> > > > > > > >
> > > > > > > > Map<String,Catalog> catalogs in CataloManager will remain
> > > > > unchanged
> > > > > > > and
> > > > > > > > save initialized catalogs.This means that deletion
> > operations
> > > > from
> > > > > > one
> > > > > > > > session won't synchronize with other sessions.
> > > > > > > > 6.
> > > > > > > >
> > > > > > > > To support multi-session synchronization scenarios for
> > > deletions
> > > > > > later
> > > > > > > > on we should make Map<String,Catalog>catalogs
> > > configurable.There
> > > > > may
> > > > > > > be
> > > > > > > > three possible situations:
> > > > > > > >
> > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > >
> > > > > > > > b.Introduction of LRU cache logic which can automatically
> > > clear
> > > > > > > > long-unused catalogs.
> > > > > > > >
> > > > > > > > c.No caching of any instances; each call to getCatalog
> > > creates a
> > > > > new
> > > > > > > > instance.
> > > > > > > >
> > > > > > > >
> > > > > > > > This is the document for discussion:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > >
> > > > > > > > This is the final proposal document:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > >
> > > > > > > >
> > > > > > > > Thank you very much for your attention and suggestions on this
> > > > > FLIP. A
> > > > > > > > special thanks to Hang Ruan for his suggestions on the entire
> > > > design
> > > > > > and
> > > > > > > > organizing offline discussions.
> > > > > > > >
> > > > > > > > If you have any further suggestions or feedback about this FLIP
> > > > > please
> > > > > > > feel
> > > > > > > > free to share.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Feng
> > > > > > > >
> > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > Thanks for improving the FLIP. It looks good to me. We could
> > > > still
> > > > > > > > > reconsider in the future how to provide more common built-in
> > > > cache
> > > > > > > > > functionality in CatalogManager so that not every
> > CatalogSotre
> > > > > > > > > implementation has to take care of it.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > jinfeng1995@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jing,
> > > > > > > > > >
> > > > > > > > > > Thanks for your reply.
> > > > > > > > > >
> > > > > > > > > > > There might be more such issues. I would suggest you
> > > > > completely
> > > > > > > walk
> > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > >
> > > > > > > > > > I am very sorry for my carelessness and at the same time, I
> > > > > greatly
> > > > > > > > > > appreciate your careful review. I have thoroughly checked
> > the
> > > > > > entire
> > > > > > > > FLIP
> > > > > > > > > > and made corrections to these issues.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > > > > CatalogManager
> > > > > > > > > > could work without Optional CatalogStore being configured.
> > > > > > > > > >
> > > > > > > > > > Yes, in the original design, CatalogStore was not necessary
> > > > > because
> > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> > > > > catalog
> > > > > > > > > > instances.
> > > > > > > > > > However, this caused inconsistency issues. Therefore, I
> > > > modified
> > > > > > this
> > > > > > > > > part
> > > > > > > > > > of the design and removed Map<String, Catalog> catalogs
> > from
> > > > > > > > > > CatalogManager.
> > > > > > > > > > At the same time, InMemoryCatalog will serve as the
> > default
> > > > > > > > CatalogStore
> > > > > > > > > > to save catalogs in memory and replace the functionality of
> > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > The previous plan that kept Map<String,Catalog>catalogs has
> > > > been
> > > > > > > moved
> > > > > > > > to
> > > > > > > > > > Rejected Alternatives.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > There are still many places contain inconsistent content,
> > > > e.g.
> > > > > > > > > > >
> > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > 2. The java comment of the method registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog catalog) in CatalogManager does not tell what the
> > > > > method
> > > > > > is
> > > > > > > > > > doing.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > There might be more such issues. I would suggest you
> > > > completely
> > > > > > > walk
> > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > >
> > > > > > > > > > > About the inMemoryCatalogStore, do you mean that you will
> > > > build
> > > > > > the
> > > > > > > > > cache
> > > > > > > > > > > functionality in the CatalogStore? This is a very
> > different
> > > > > > design
> > > > > > > > > > concept
> > > > > > > > > > > from what the current FLIP described. If I am not
> > mistaken,
> > > > > with
> > > > > > > the
> > > > > > > > > > > current FLIP design, CatalogManager could work without
> > > > Optional
> > > > > > > > > > > CatalogStore being configured. That is the reason why I
> > > > > mentioned
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > last email that the example code wrt the Optional
> > > > CatalogStore
> > > > > is
> > > > > > > not
> > > > > > > > > > > correct.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > jinfeng1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jing
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you very much for the careful review. I have
> > > already
> > > > > made
> > > > > > > the
> > > > > > > > > > > > relevant changes.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > Speaking of the conflict issues in the
> > multi-instance
> > > > > > > > scenarios, I
> > > > > > > > > > am
> > > > > > > > > > > > not
> > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > >
> > > > > > > > > > > > Currently, there are conflicts in multiple scenarios
> > with
> > > > the
> > > > > > > > current
> > > > > > > > > > > > design. I am thinking whether we should remove
> > > 'Map<String,
> > > > > > > > Catalog>'
> > > > > > > > > > and
> > > > > > > > > > > > make Cache the default behavior of
> > InMemoryCatalogStore.
> > > > This
> > > > > > > way,
> > > > > > > > > > users
> > > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > > multi-instance
> > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for working on the FLIP. There are still some
> > > NIT
> > > > > > issues
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > like:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used
> > as
> > > > > > > > > CatalogStore
> > > > > > > > > > > > > instead of Optional in the code example. It should be
> > > > fine
> > > > > to
> > > > > > > use
> > > > > > > > > it
> > > > > > > > > > as
> > > > > > > > > > > > > pseudo code for now and update it after you submit
> > the
> > > > PR.
> > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> > > > rejected
> > > > > > > > section
> > > > > > > > > > > which
> > > > > > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > > > scenarios,
> > > > > > > > I
> > > > > > > > > am
> > > > > > > > > > > not
> > > > > > > > > > > > > sure if this is the intended behaviour. If
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs
> > > > > > > > > > > > is
> > > > > > > > > > > > > used as a cache, it should be invalid, once the
> > related
> > > > > > catalog
> > > > > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > > > removed from the CatalogStore by another instance.
> > Did
> > > I
> > > > > miss
> > > > > > > > > > > something?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > I think this is a good function name, I have
> > updated
> > > it
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Some common cache features should be
> > implemented
> > > > > > > > > > > > > > Thank you for the suggestion. If alternative 1
> > turns
> > > > out
> > > > > to
> > > > > > > be
> > > > > > > > > more
> > > > > > > > > > > > > > appropriate later, I will improve this part of the
> > > > > content.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As the above discussion moves forward, the
> > option 2
> > > > > > > solution
> > > > > > > > > > looks
> > > > > > > > > > > > more
> > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > Yes, after discussing with Shammon offline, we
> > think
> > > > that
> > > > > > > > > solution
> > > > > > > > > > 2
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > be more suitable and also avoid any inconsistency
> > > > issues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are some inconsistent descriptions in the
> > > > > content.
> > > > > > > > Would
> > > > > > > > > > you
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > I will do my best to improve the document and
> > > > appreciate
> > > > > > your
> > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > can you put the unselected option in `Rejected
> > > > > > > Alternatives`
> > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > Alternatives`.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > > > zjureel@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I found there are two options in `Proposed
> > > Changes`,
> > > > > can
> > > > > > > you
> > > > > > > > > put
> > > > > > > > > > > the
> > > > > > > > > > > > > > > unselected option in `Rejected Alternatives`? I
> > > think
> > > > > > this
> > > > > > > > may
> > > > > > > > > > help
> > > > > > > > > > > > us
> > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am still
> > confused
> > > > > after
> > > > > > > > > > > completely
> > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > > > > addCatalog()
> > > > > > > > have
> > > > > > > > > > no
> > > > > > > > > > > > big
> > > > > > > > > > > > > > > > difference based on their names. One of them is
> > > > > > > responsible
> > > > > > > > > for
> > > > > > > > > > > > data
> > > > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> > > > > catalogs
> > > > > > is
> > > > > > > > > used
> > > > > > > > > > > as a
> > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > and catalogStore is used for data persistence.
> > I
> > > > > would
> > > > > > > > > suggest
> > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > their purpose conceptually and clearly in the
> > > FLIP.
> > > > > > Some
> > > > > > > > > common
> > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > features should be implemented, i.e. data in
> > the
> > > > > cache
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > store
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > be consistent. Same Catalog instance should be
> > > > found
> > > > > in
> > > > > > > the
> > > > > > > > > > store
> > > > > > > > > > > > and
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the cache(either it has been initialized or it
> > > will
> > > > > be
> > > > > > > lazy
> > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > for the same catalog name. The consistency will
> > > be
> > > > > > taken
> > > > > > > > care
> > > > > > > > > > of
> > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > 3. As the above discussion moves forward, the
> > > > option
> > > > > 2
> > > > > > > > > solution
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> > > > > issues
> > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > previously with option 1 are not solved yet. Do
> > > you
> > > > > > still
> > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > both options and ask for suggestions for both
> > of
> > > > > them?
> > > > > > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > > > > > inconsistent
> > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the content. Would you like to clean them up?
> > > > > Thanks!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you for your response, and I completely
> > > > agree
> > > > > > > with
> > > > > > > > > your
> > > > > > > > > > > > point
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > Initially, I may have over complicated the
> > > whole
> > > > > > issue.
> > > > > > > > > First
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > we need to consider the persistence of the
> > > > > Catalog's
> > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > If we only need to provide persistence for
> > > > Catalog
> > > > > > > > > > > Configuration,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > > > > > interface.
> > > > > > > > > > > > > > > > > This method can convert a Catalog instance
> > to a
> > > > > > > > Map<String,
> > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > properties, and the default implementation
> > will
> > > > > throw
> > > > > > > an
> > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > * Returns a map containing the properties
> > of
> > > > the
> > > > > > > > catalog
> > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > * @return a map containing the properties
> > of
> > > > the
> > > > > > > > catalog
> > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > * @throws UnsupportedOperationException if
> > > the
> > > > > > > > > > implementing
> > > > > > > > > > > > > class
> > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > * the default implementation of this
> > method
> > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > default Map<String, String> toProperties()
> > {
> > > > > > > > > > > > > > > > > throw new
> > > > UnsupportedOperationException("Please
> > > > > > > > > implement
> > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. If the user has configured a
> > CatalogStore,
> > > > the
> > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > will be called when registering a Catalog
> > > > instance
> > > > > > with
> > > > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > > > catalog).
> > > > > > > The
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > will be converted to a Map<String, String>
> > > > > properties
> > > > > > > and
> > > > > > > > > > saved
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore. If some Catalog instances do
> > not
> > > > > > > implement
> > > > > > > > > this
> > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. If the user does not use a CatalogStore,
> > > the
> > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > will not be called, ensuring consistency with
> > > the
> > > > > > > > original
> > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> > > catalogs
> > > > > and
> > > > > > > the
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > we can start from the Catalog itself and
> > > provide
> > > > a
> > > > > > > > > dedicated
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > public class ConfigurationCatalog implements
> > > > > Catalog
> > > > > > {
> > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > > > > properties)
> > > > > > {
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I have no questions about the functionality
> > > of
> > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > `registerCatalog`
> > > > > > or
> > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > `CatalogManager`. The implementation in
> > Trino
> > > > is
> > > > > > also
> > > > > > > > the
> > > > > > > > > > > same,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > > > > `createCatalog`,
> > > > > > > > > > which
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > > > `registerCatalog`
> > > > > > > > or
> > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> > > > catalogs`
> > > > > > or
> > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned before,
> > the
> > > > > > > > `Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`.
> > How
> > > > > about
> > > > > > > > > saving
> > > > > > > > > > > the
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > > `CatalogStore
> > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thank you for your reply. I am very sorry
> > > for
> > > > > the
> > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > by my deviation from the original
> > > discussion.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1]
> > for
> > > > > this
> > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > Yes, there was indeed such a discussion
> > > > before.
> > > > > > > > > However,
> > > > > > > > > > > in
> > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > > > > > CatalogManager
> > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > need to change much. *We cannot only
> > > persist
> > > > > > > Catalog
> > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > not be enough to save Catalogs, because
> > in
> > > > the
> > > > > > end
> > > > > > > we
> > > > > > > > > > still
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > the configurations corresponding to the
> > > > Catalog
> > > > > > > > > > instances.
> > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> > > > interface
> > > > > > for
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > persistence. Regarding this part, I also
> > > took
> > > > > > > > > inspiration
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> > with
> > > > the
> > > > > > > same
> > > > > > > > > name
> > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I will
> > > provide
> > > > a
> > > > > > > > detailed
> > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > code examples for this part, and add it
> > to
> > > > the
> > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > > still
> > > > > > looks
> > > > > > > > > > confused
> > > > > > > > > > > > to
> > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > CatalogStore,
> > > > > > > > how
> > > > > > > > > > > about
> > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > opinion,
> > > > > > you
> > > > > > > > can
> > > > > > > > > > > decide
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me.
> > I
> > > > > don't
> > > > > > > see
> > > > > > > > > any
> > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which
> > > > method
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > > > wondering
> > > > > > > > > whether
> > > > > > > > > > > user
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original design
> > was
> > > to
> > > > > add
> > > > > > > it
> > > > > > > > > > > directly
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> > > > > existing
> > > > > > > > > logic.
> > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > we can do both: save the configuration to
> > > the
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> > > > > default
> > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > I think it is necessary, otherwise we
> > would
> > > > > need
> > > > > > to
> > > > > > > > > > > introduce
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > Map to store the configuration for lazy
> > > > > loading.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > > > catalog.
> > > > > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > made into an asynchronous interface
> > because
> > > > > > Catalog
> > > > > > > > is
> > > > > > > > > > > > already
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> > > for
> > > > > > > catalog
> > > > > > > > > than
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> > > > > should
> > > > > > be
> > > > > > > > > lazy
> > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > instead of asynchronous registration. I
> > > have
> > > > > > > already
> > > > > > > > > > > updated
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon
> > FY <
> > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> > > > > method
> > > > > > in
> > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > confuse users whether a method named
> > > > > > `addCatalog`
> > > > > > > > or
> > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > And as you mentioned, the memory
> > catalog
> > > > is a
> > > > > > > > > `cache`,
> > > > > > > > > > I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > of `cache` should not be exposed to
> > > users.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1]
> > for
> > > > > this
> > > > > > > > FLIP.
> > > > > > > > > > > Please
> > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > > > > > discussion
> > > > > > > > is
> > > > > > > > > > to
> > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> > > > > implement
> > > > > > it
> > > > > > > > for
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think there is a gap between the
> > > current
> > > > > FLIP
> > > > > > > > > design
> > > > > > > > > > > and
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > What about the proposal of the
> > discussion
> > > > in
> > > > > > > thread
> > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > >
> > > > > > > > > >
> > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> > > with
> > > > > the
> > > > > > > > same
> > > > > > > > > > name
> > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > complex to me. I think it'll be
> > better
> > > to
> > > > > > > explain
> > > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > these methods and give a brief
> > example
> > > in
> > > > > > this
> > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > > > still
> > > > > > > looks
> > > > > > > > > > > confused
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > > > CatalogStore,
> > > > > > > > > > how
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > > > opinion,
> > > > > > > > you
> > > > > > > > > > can
> > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement,
> > which
> > > > > method
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`?
> > I'm
> > > > > > > wondering
> > > > > > > > > > > whether
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> > provide a
> > > > > > default
> > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 4: About asynchronous registration
> > for
> > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > > > > > CATALOG,
> > > > > > > > the
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration method is used by
> > default.
> > > > > > > > > > > > > > > > > > > > > If asynchronous registration is the
> > > > default
> > > > > > > > > behavior,
> > > > > > > > > > > it
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > > > > > registration
> > > > > > > > > just
> > > > > > > > > > > > like
> > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > > > > `registerCatalog`
> > > > > > > > > > be
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous registration,
> > it
> > > > may
> > > > > > well
> > > > > > > > > that
> > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > executes successfully, but then the
> > > > > following
> > > > > > > > > CREATE
> > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > > > > > I think it's a break change which
> > > should
> > > > be
> > > > > > > > > > highlighted
> > > > > > > > > > > > in
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > registration, I
> > > > > > > would
> > > > > > > > > > like
> > > > > > > > > > > to
> > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > will be an executor to open or
> > register
> > > > > > catalog
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > > initialized
> > > > > if
> > > > > > it
> > > > > > > > has
> > > > > > > > > > not
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > yet. If the initialization process
> > > fails,
> > > > > > these
> > > > > > > > > > > > statements
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > initialization
> > > > for
> > > > > > > > catalog
> > > > > > > > > > than
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > jinfeng1995@gmail.com
> > > >
> > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > > > > > persistence
> > > > > > > > of
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> > > > > interest
> > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > How to handle a catalog with the
> > same
> > > > > name
> > > > > > > that
> > > > > > > > > > > exists
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I believe this is a crucial point.
> > > Based
> > > > on
> > > > > > my
> > > > > > > > > > current
> > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > understanding, the Map<String,
> > Catalog>
> > > > > > > catalogs
> > > > > > > > > will
> > > > > > > > > > > > serve
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > for instantiated catalogs and have
> > the
> > > > > > highest
> > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There are three methods that can
> > affect
> > > > the
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This method puts the catalog instance
> > > > into
> > > > > > the
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > catalogName)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This method removes the catalog
> > > instance
> > > > > > > > > > corresponding
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> > catalogs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > > > corresponding
> > > > > > > > > catalog
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If
> > > the
> > > > > > > catalog
> > > > > > > > > > does
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > configuration
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > > initialized
> > > > > > > Catalog
> > > > > > > > > > > instance
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The following two methods only modify
> > > the
> > > > > > > > > > configuration
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > > > > Map<String,
> > > > > > > > > String>
> > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This method saves the properties to
> > the
> > > > > > > > > catalogStore
> > > > > > > > > > > and
> > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > whether there is a catalogName with
> > the
> > > > > same
> > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > > > This method removes the specified
> > > > > > configuration
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The following are possible conflict
> > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> > > > > already
> > > > > > > > > exists
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > Map<String,
> > > > > > > Catalog>,
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > > method
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > > > > > Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> > > > > already
> > > > > > > > > exists
> > > > > > > > > > in
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> > > > Catalog>,
> > > > > > the
> > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method
> > > will
> > > > > > fail.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> > > > > already
> > > > > > > > > exists
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > addCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > String> properties) method can
> > directly
> > > > > save
> > > > > > > the
> > > > > > > > > > > > properties
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > getCatalog(String
> > > > > > > > > catalogName)
> > > > > > > > > > > > method
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > initialization
> > > > > > > because
> > > > > > > > > the
> > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > catalog instance already exists in
> > > > catalogs
> > > > > > and
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > unregisterCatalog(String
> > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > remove the instance corresponding to
> > > the
> > > > > > > original
> > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > > > > > `Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > > > > > Map<String,
> > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This could potentially lead to
> > > > confusion. I
> > > > > > > > suggest
> > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > addCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > String> properties), as previously
> > > > > mentioned
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > > catalogName,Catalog
> > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> > instance,
> > > > > > adding
> > > > > > > > the
> > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > parameter to the
> > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > method may not necessarily result in
> > > lazy
> > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > encryption
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think encryption is necessary, but
> > > > > perhaps
> > > > > > > the
> > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > should be implemented in the
> > > CatalogStore
> > > > > > when
> > > > > > > > the
> > > > > > > > > > data
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > FileCatalogStore
> > > > > > could
> > > > > > > > > > encrypt
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > > > > MemoryCatalogStore
> > > > > > > > > > > does
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > > > > MemoryCatalogStore
> > > > > > > > > > as
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > > > > > configurations
> > > > > > > > > in
> > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > > > > > asynchronous
> > > > > > > > > > loading
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > would need to introduce additional
> > > cache.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > What if asynchronous registration
> > > > failed?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This is also a critical concern.
> > When
> > > > > > > executing
> > > > > > > > > DDL,
> > > > > > > > > > > > DQL,
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > statements that reference a specific
> > > > > Catalog,
> > > > > > > the
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > will be initialized if it has not
> > been
> > > > > > > > initialized
> > > > > > > > > > yet.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > initialization process fails, these
> > > > > > statements
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog>
> > catalogs
> > > > and
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the CatalogManager all have the same
> > > > > catalog
> > > > > > > > name,
> > > > > > > > > > but
> > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> > referring
> > > > to
> > > > > > the
> > > > > > > > > > previous
> > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > key principle is that the Catalog
> > that
> > > > has
> > > > > > been
> > > > > > > > > most
> > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > will be given priority for subsequent
> > > > use.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM
> > Xianxun
> > > > Ye
> > > > > <
> > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> > > > > believe
> > > > > > > this
> > > > > > > > > > Flip
> > > > > > > > > > > > > could
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > for users. The firm I work for also
> > > > > manages a
> > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > submission of tasks becomes slow
> > > because
> > > > of
> > > > > > > > > loading a
> > > > > > > > > > > > > number
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the user's
> > > > Flink
> > > > > > SQL
> > > > > > > > > > through
> > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs to
> > > improve
> > > > > the
> > > > > > > > speed
> > > > > > > > > of
> > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog with
> > > CREATE
> > > > > > > CATALOG,
> > > > > > > > > the
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration method is used by
> > default.
> > > > > What
> > > > > > if
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > > asynchronous
> > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. I also have the same question
> > with
> > > > > > > Shammon’s
> > > > > > > > > > first
> > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > > > > CatalogStore
> > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogManager all have the same
> > > catalog
> > > > > > name,
> > > > > > > > but
> > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > different catalog instances. For
> > > example:
> > > > > > > > > catalog1->
> > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog
> > > in
> > > > > the
> > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> > > `catalogs`
> > > > > and
> > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi jark, thanks for your suggestion.
> 1. How to register the CatalogStore for Table API? I think the
CatalogStore should be immutable once TableEnv is created. Otherwise, there
might be some data inconsistencies when CatalogStore is changed.
Yes, We should initialize the CatalogStore when creating the TableEnv.
Therefore, my current proposal is to add a method to configure the
CatalogStore in EnvironmentSettings.
// Initialize a catalog Store
CatalogStore catalogStore = new FileCatalogStore("");
// set up the Table API
final EnvironmentSettings settings =
EnvironmentSettings.newInstance().inBatchMode()
.withCatalogStore(catalogStore)
.build();
final TableEnvironment tableEnv = TableEnvironment.create(settings);
> 2. Why does the CatalogStoreFactory interface only have a default method,
not an interface method?
Sorry, While I did refer to the Catalog interface, I agree that as a new
interface, the CatalogStoreFactory should not have a default method but an
interface method. I have already modified the interface.
> 3. Please mention the alternative API in Javadoc for the
deprecated`registerCatalog`.
> 4. In the "Compatibility" section, would be better to mention the changed
behavior of CREATE CATALOG statement if FileCatalogStore (or other
persisted catalog store) is used.
Thanks for the suggestion, I have updated the FLIP [1].
[1].
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
Best,
Feng
On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> Hi Feng,
>
> This is a useful FLIP. Thanks for starting this discussion.
> The current design looks pretty good to me. I just have some minor
> comments.
>
> 1. How to register the CatalogStore for Table API? I think the CatalogStore
> should be immutable once TableEnv is created. Otherwise, there might be
> some data inconsistencies when CatalogStore is changed.
>
> 2. Why does the CatalogStoreFactory interface only have a default method,
> not an interface method?
>
> 3. Please mention the alternative API in Javadoc for the deprecated
> `registerCatalog`.
>
> 4. In the "Compatibility" section, would be better to mention the changed
> behavior of CREATE CATALOG statement if FileCatalogStore (or other
> persisted catalog store) is used.
>
>
> Best,
> Jark
>
> On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
>
> > Hi ， thanks all for reviewing the flip.
> >
> > @Ron
> >
> > > Regarding the CatalogStoreFactory#createCatalogStore method, do we
> need
> > to provide a default implementation?
> >
> > Yes, we will provide a default InMemoryCatalogStoreFactory to create an
> > InMemoryCatalogStore.
> >
> > > If we get a Catalog from CatalogStore, after initializing it, whether
> we
> > put it in Map<String, Catalog> catalogs again?
> >
> > Yes, in the current design, catalogs are stored as snapshots, and once
> > initialized, the Catalog will be placed in the Map<String, Catalog>
> > catalogs.
> > Subsequently, the Map<String, Catalog> catalogs will be the primary
> source
> > for obtaining the corresponding Catalog.
> >
> > > how about renaming them to `catalog.store.type` and
> > `catalog.store.path`?
> >
> > I think it is okay. Adding "sql" at the beginning may seem a bit
> strange. I
> > will update the FLIP.
> >
> >
> >
> > @Shammon
> >
> > Thank you for the review. I have made the necessary corrections.
> > Regarding the modifications made to the Public Interface, I have also
> > included the relevant changes to the `TableEnvironment`.
> >
> >
> > Best,
> > Feng
> >
> >
> > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com> wrote:
> >
> > > Hi feng,
> > >
> > > Thanks for updating, I have some minor comments
> > >
> > > 1. The modification of `CatalogManager` should not be in `Public
> > > Interfaces`, it is not a public interface.
> > >
> > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > `CatalogStoreFactory`
> > >
> > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > catalogStore.get(catalogName);`
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
> > >
> > > > Hi, Feng
> > > >
> > > > Thanks for driving this FLIP, this proposal is very useful for
> catalog
> > > > management.
> > > > I have some small questions:
> > > >
> > > > 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > > need
> > > > to provide a default implementation?
> > > > 2. If we get Catalog from CatalogStore, after initializing it,
> whether
> > we
> > > > put it to Map<String, Catalog> catalogs again?
> > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > `catalog.store.type` and `catalog.store.path`?
> > > >
> > > > Best,
> > > > Ron
> > > >
> > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > >
> > > > > Hi yuxia
> > > > >
> > > > > > But from the code in Proposed Changes, once we register the
> > Catalog,
> > > > we
> > > > > initialize it and open it. right?
> > > > >
> > > > > Yes, In order to avoid inconsistent semantics of the original
> CREATE
> > > > > CATALOG DDL, Catalog will be directly initialized in
> registerCatalog
> > so
> > > > > that parameter validation can be performed.
> > > > >
> > > > > In the current design, lazy initialization is mainly reflected in
> > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > configurations,
> > > > > only initialization is required in getCatalog.
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <luoyuxia@alumni.sjtu.edu.cn
> >
> > > > wrote:
> > > > >
> > > > > > Hi, Feng.
> > > > > > I'm trying to understanding the meaning of *lazy initialization*.
> > If
> > > > i'm
> > > > > > wrong, please correct me.
> > > > > >
> > > > > > IIUC, lazy initialization means only you need to access the
> > catalog,
> > > > then
> > > > > > you initialize it. But from the code in Proposed Changes, once we
> > > > > register
> > > > > > the Catalog,
> > > > > > we initialize it and open it. right?
> > > > > >
> > > > > > Best regards,
> > > > > > Yuxia
> > > > > >
> > > > > > ----- 原始邮件 -----
> > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > configuration
> > > > > > and asynchronous registration
> > > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > Thanks for your effort! +1 for the proposal.
> > > > > >
> > > > > > One of the major changes is that current design will provide
> > > > > > Map<String,Catalog> catalogs as a snapshot instead of a cache,
> > which
> > > > > means
> > > > > > once it has been initialized, any changes done by other sessions
> > will
> > > > not
> > > > > > affect it. Point 6 described follow-up options for further
> > > improvement.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi all, I would like to update you on the latest progress of
> the
> > > > FLIP.
> > > > > > >
> > > > > > >
> > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai
> > Fang
> > > > > and I
> > > > > > > had an offline discussion regarding the overall solution for
> > Flink
> > > > > > > CatalogStore. We have reached a consensus and I have updated
> the
> > > > final
> > > > > > > solution in FLIP.
> > > > > > >
> > > > > > > Next, let me briefly describe the entire design:
> > > > > > >
> > > > > > > 1.
> > > > > > >
> > > > > > > Introduce CatalogDescriptor to store catalog configuration
> > > similar
> > > > > to
> > > > > > > TableDescriptor.
> > > > > > > 2.
> > > > > > >
> > > > > > > The two key functions of CatalogStore - void
> > storeCatalog(String
> > > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > > getCatalog(String)
> > > > > > > will both use CatalogDescriptor instead of Catalog instance.
> > > This
> > > > > way,
> > > > > > > CatalogStore will only be responsible for saving and
> > retrieving
> > > > > > catalog
> > > > > > > configurations without having to initialize catalogs.
> > > > > > > 3.
> > > > > > >
> > > > > > > The default registerCatalog(String catalogName, Catalog
> > catalog)
> > > > > > > function in CatalogManager will be marked as deprecated.
> > > > > > > 4.
> > > > > > >
> > > > > > > A new function registerCatalog(String catalogName,
> > > > CatalogDescriptor
> > > > > > > catalog) will be added to serve as the default registration
> > > > function
> > > > > > for
> > > > > > > catalogs in CatalogManager.
> > > > > > > 5.
> > > > > > >
> > > > > > > Map<String,Catalog> catalogs in CataloManager will remain
> > > > unchanged
> > > > > > and
> > > > > > > save initialized catalogs.This means that deletion
> operations
> > > from
> > > > > one
> > > > > > > session won't synchronize with other sessions.
> > > > > > > 6.
> > > > > > >
> > > > > > > To support multi-session synchronization scenarios for
> > deletions
> > > > > later
> > > > > > > on we should make Map<String,Catalog>catalogs
> > configurable.There
> > > > may
> > > > > > be
> > > > > > > three possible situations:
> > > > > > >
> > > > > > > a.Default caching of all initialized catalogs
> > > > > > >
> > > > > > > b.Introduction of LRU cache logic which can automatically
> > clear
> > > > > > > long-unused catalogs.
> > > > > > >
> > > > > > > c.No caching of any instances; each call to getCatalog
> > creates a
> > > > new
> > > > > > > instance.
> > > > > > >
> > > > > > >
> > > > > > > This is the document for discussion:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > >
> > > > > > > This is the final proposal document:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > >
> > > > > > >
> > > > > > > Thank you very much for your attention and suggestions on this
> > > > FLIP. A
> > > > > > > special thanks to Hang Ruan for his suggestions on the entire
> > > design
> > > > > and
> > > > > > > organizing offline discussions.
> > > > > > >
> > > > > > > If you have any further suggestions or feedback about this FLIP
> > > > please
> > > > > > feel
> > > > > > > free to share.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Feng
> > > > > > >
> > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > Thanks for improving the FLIP. It looks good to me. We could
> > > still
> > > > > > > > reconsider in the future how to provide more common built-in
> > > cache
> > > > > > > > functionality in CatalogManager so that not every
> CatalogSotre
> > > > > > > > implementation has to take care of it.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> jinfeng1995@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jing,
> > > > > > > > >
> > > > > > > > > Thanks for your reply.
> > > > > > > > >
> > > > > > > > > > There might be more such issues. I would suggest you
> > > > completely
> > > > > > walk
> > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > >
> > > > > > > > > I am very sorry for my carelessness and at the same time, I
> > > > greatly
> > > > > > > > > appreciate your careful review. I have thoroughly checked
> the
> > > > > entire
> > > > > > > FLIP
> > > > > > > > > and made corrections to these issues.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > > > CatalogManager
> > > > > > > > > could work without Optional CatalogStore being configured.
> > > > > > > > >
> > > > > > > > > Yes, in the original design, CatalogStore was not necessary
> > > > because
> > > > > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> > > > catalog
> > > > > > > > > instances.
> > > > > > > > > However, this caused inconsistency issues. Therefore, I
> > > modified
> > > > > this
> > > > > > > > part
> > > > > > > > > of the design and removed Map<String, Catalog> catalogs
> from
> > > > > > > > > CatalogManager.
> > > > > > > > > At the same time, InMemoryCatalog will serve as the
> default
> > > > > > > CatalogStore
> > > > > > > > > to save catalogs in memory and replace the functionality of
> > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > The previous plan that kept Map<String,Catalog>catalogs has
> > > been
> > > > > > moved
> > > > > > > to
> > > > > > > > > Rejected Alternatives.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > There are still many places contain inconsistent content,
> > > e.g.
> > > > > > > > > >
> > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > 2. The java comment of the method registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog catalog) in CatalogManager does not tell what the
> > > > method
> > > > > is
> > > > > > > > > doing.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > There might be more such issues. I would suggest you
> > > completely
> > > > > > walk
> > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > >
> > > > > > > > > > About the inMemoryCatalogStore, do you mean that you will
> > > build
> > > > > the
> > > > > > > > cache
> > > > > > > > > > functionality in the CatalogStore? This is a very
> different
> > > > > design
> > > > > > > > > concept
> > > > > > > > > > from what the current FLIP described. If I am not
> mistaken,
> > > > with
> > > > > > the
> > > > > > > > > > current FLIP design, CatalogManager could work without
> > > Optional
> > > > > > > > > > CatalogStore being configured. That is the reason why I
> > > > mentioned
> > > > > > in
> > > > > > > > the
> > > > > > > > > > last email that the example code wrt the Optional
> > > CatalogStore
> > > > is
> > > > > > not
> > > > > > > > > > correct.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > jinfeng1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jing
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > >
> > > > > > > > > > > Thank you very much for the careful review. I have
> > already
> > > > made
> > > > > > the
> > > > > > > > > > > relevant changes.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > Speaking of the conflict issues in the
> multi-instance
> > > > > > > scenarios, I
> > > > > > > > > am
> > > > > > > > > > > not
> > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > >
> > > > > > > > > > > Currently, there are conflicts in multiple scenarios
> with
> > > the
> > > > > > > current
> > > > > > > > > > > design. I am thinking whether we should remove
> > 'Map<String,
> > > > > > > Catalog>'
> > > > > > > > > and
> > > > > > > > > > > make Cache the default behavior of
> InMemoryCatalogStore.
> > > This
> > > > > > way,
> > > > > > > > > users
> > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > multi-instance
> > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for working on the FLIP. There are still some
> > NIT
> > > > > issues
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > FLIP
> > > > > > > > > > > > like:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used
> as
> > > > > > > > CatalogStore
> > > > > > > > > > > > instead of Optional in the code example. It should be
> > > fine
> > > > to
> > > > > > use
> > > > > > > > it
> > > > > > > > > as
> > > > > > > > > > > > pseudo code for now and update it after you submit
> the
> > > PR.
> > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> > > rejected
> > > > > > > section
> > > > > > > > > > which
> > > > > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > > > > >
> > > > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > > scenarios,
> > > > > > > I
> > > > > > > > am
> > > > > > > > > > not
> > > > > > > > > > > > sure if this is the intended behaviour. If
> Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs
> > > > > > > > > > > is
> > > > > > > > > > > > used as a cache, it should be invalid, once the
> related
> > > > > catalog
> > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > removed from the CatalogStore by another instance.
> Did
> > I
> > > > miss
> > > > > > > > > > something?
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > I think this is a good function name, I have
> updated
> > it
> > > > in
> > > > > > the
> > > > > > > > > > > > > documentation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Some common cache features should be
> implemented
> > > > > > > > > > > > > Thank you for the suggestion. If alternative 1
> turns
> > > out
> > > > to
> > > > > > be
> > > > > > > > more
> > > > > > > > > > > > > appropriate later, I will improve this part of the
> > > > content.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > As the above discussion moves forward, the
> option 2
> > > > > > solution
> > > > > > > > > looks
> > > > > > > > > > > more
> > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > Yes, after discussing with Shammon offline, we
> think
> > > that
> > > > > > > > solution
> > > > > > > > > 2
> > > > > > > > > > > > might
> > > > > > > > > > > > > be more suitable and also avoid any inconsistency
> > > issues.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > There are some inconsistent descriptions in the
> > > > content.
> > > > > > > Would
> > > > > > > > > you
> > > > > > > > > > > > like
> > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > I will do my best to improve the document and
> > > appreciate
> > > > > your
> > > > > > > > > > > > suggestions.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > can you put the unselected option in `Rejected
> > > > > > Alternatives`
> > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> Alternatives`.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > > zjureel@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I found there are two options in `Proposed
> > Changes`,
> > > > can
> > > > > > you
> > > > > > > > put
> > > > > > > > > > the
> > > > > > > > > > > > > > unselected option in `Rejected Alternatives`? I
> > think
> > > > > this
> > > > > > > may
> > > > > > > > > help
> > > > > > > > > > > us
> > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for raising this FLIP. I am still
> confused
> > > > after
> > > > > > > > > > completely
> > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > > > addCatalog()
> > > > > > > have
> > > > > > > > > no
> > > > > > > > > > > big
> > > > > > > > > > > > > > > difference based on their names. One of them is
> > > > > > responsible
> > > > > > > > for
> > > > > > > > > > > data
> > > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> > > > catalogs
> > > > > is
> > > > > > > > used
> > > > > > > > > > as a
> > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > and catalogStore is used for data persistence.
> I
> > > > would
> > > > > > > > suggest
> > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > their purpose conceptually and clearly in the
> > FLIP.
> > > > > Some
> > > > > > > > common
> > > > > > > > > > > cache
> > > > > > > > > > > > > > > features should be implemented, i.e. data in
> the
> > > > cache
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > store
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > be consistent. Same Catalog instance should be
> > > found
> > > > in
> > > > > > the
> > > > > > > > > store
> > > > > > > > > > > and
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the cache(either it has been initialized or it
> > will
> > > > be
> > > > > > lazy
> > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > for the same catalog name. The consistency will
> > be
> > > > > taken
> > > > > > > care
> > > > > > > > > of
> > > > > > > > > > > > while
> > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > 3. As the above discussion moves forward, the
> > > option
> > > > 2
> > > > > > > > solution
> > > > > > > > > > > looks
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> > > > issues
> > > > > > > > > mentioned
> > > > > > > > > > > > > > > previously with option 1 are not solved yet. Do
> > you
> > > > > still
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > both options and ask for suggestions for both
> of
> > > > them?
> > > > > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > > > > inconsistent
> > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the content. Would you like to clean them up?
> > > > Thanks!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you for your response, and I completely
> > > agree
> > > > > > with
> > > > > > > > your
> > > > > > > > > > > point
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > Initially, I may have over complicated the
> > whole
> > > > > issue.
> > > > > > > > First
> > > > > > > > > > and
> > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > we need to consider the persistence of the
> > > > Catalog's
> > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > If we only need to provide persistence for
> > > Catalog
> > > > > > > > > > Configuration,
> > > > > > > > > > > > we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > > > > interface.
> > > > > > > > > > > > > > > > This method can convert a Catalog instance
> to a
> > > > > > > Map<String,
> > > > > > > > > > > String>
> > > > > > > > > > > > > > > > properties, and the default implementation
> will
> > > > throw
> > > > > > an
> > > > > > > > > > > exception.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > * Returns a map containing the properties
> of
> > > the
> > > > > > > catalog
> > > > > > > > > > > object.
> > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > * @return a map containing the properties
> of
> > > the
> > > > > > > catalog
> > > > > > > > > > > object
> > > > > > > > > > > > > > > > * @throws UnsupportedOperationException if
> > the
> > > > > > > > > implementing
> > > > > > > > > > > > class
> > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > * the default implementation of this
> method
> > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > default Map<String, String> toProperties()
> {
> > > > > > > > > > > > > > > > throw new
> > > UnsupportedOperationException("Please
> > > > > > > > implement
> > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. If the user has configured a
> CatalogStore,
> > > the
> > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > will be called when registering a Catalog
> > > instance
> > > > > with
> > > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > > catalog).
> > > > > > The
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > will be converted to a Map<String, String>
> > > > properties
> > > > > > and
> > > > > > > > > saved
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore. If some Catalog instances do
> not
> > > > > > implement
> > > > > > > > this
> > > > > > > > > > > > method,
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. If the user does not use a CatalogStore,
> > the
> > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > will not be called, ensuring consistency with
> > the
> > > > > > > original
> > > > > > > > > > > process.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> > catalogs
> > > > and
> > > > > > the
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > we can start from the Catalog itself and
> > provide
> > > a
> > > > > > > > dedicated
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > public class ConfigurationCatalog implements
> > > > Catalog
> > > > > {
> > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > > > properties)
> > > > > {
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have no questions about the functionality
> > of
> > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > different operations of multiple
> > > > `registerCatalog`
> > > > > or
> > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > `CatalogManager`. The implementation in
> Trino
> > > is
> > > > > also
> > > > > > > the
> > > > > > > > > > same,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > > > `createCatalog`,
> > > > > > > > > which
> > > > > > > > > > > > will
> > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > > `registerCatalog`
> > > > > > > or
> > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> > > catalogs`
> > > > > or
> > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > catalogStore`. As you mentioned before,
> the
> > > > > > > `Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`.
> How
> > > > about
> > > > > > > > saving
> > > > > > > > > > the
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > `CatalogStore
> > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > when it is registered or added in
> > > > `CatalogManager`?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you for your reply. I am very sorry
> > for
> > > > the
> > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > by my deviation from the original
> > discussion.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1]
> for
> > > > this
> > > > > > FLIP
> > > > > > > > > > > > > > > > > > Yes, there was indeed such a discussion
> > > before.
> > > > > > > > However,
> > > > > > > > > > in
> > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > > > > CatalogManager
> > > > > > > > > > > itself
> > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > need to change much. *We cannot only
> > persist
> > > > > > Catalog
> > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > not be enough to save Catalogs, because
> in
> > > the
> > > > > end
> > > > > > we
> > > > > > > > > still
> > > > > > > > > > > > need
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > the configurations corresponding to the
> > > Catalog
> > > > > > > > > instances.
> > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> > > interface
> > > > > for
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > persistence. Regarding this part, I also
> > took
> > > > > > > > inspiration
> > > > > > > > > > > from
> > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> with
> > > the
> > > > > > same
> > > > > > > > name
> > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > Thank you for the suggestion. I will
> > provide
> > > a
> > > > > > > detailed
> > > > > > > > > > > > > description
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > code examples for this part, and add it
> to
> > > the
> > > > > > FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > still
> > > > > looks
> > > > > > > > > confused
> > > > > > > > > > > to
> > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > CatalogStore,
> > > > > > > how
> > > > > > > > > > about
> > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > opinion,
> > > > > you
> > > > > > > can
> > > > > > > > > > decide
> > > > > > > > > > > > to
> > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me.
> I
> > > > don't
> > > > > > see
> > > > > > > > any
> > > > > > > > > > > > problem
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which
> > > method
> > > > > will
> > > > > > > be
> > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > > wondering
> > > > > > > > whether
> > > > > > > > > > user
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original design
> was
> > to
> > > > add
> > > > > > it
> > > > > > > > > > directly
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> > > > existing
> > > > > > > > logic.
> > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > we can do both: save the configuration to
> > the
> > > > > > > > > CatalogStore
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> > > > default
> > > > > > > > > > > implmentation
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > I think it is necessary, otherwise we
> would
> > > > need
> > > > > to
> > > > > > > > > > introduce
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > Map to store the configuration for lazy
> > > > loading.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > > catalog.
> > > > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > made into an asynchronous interface
> because
> > > > > Catalog
> > > > > > > is
> > > > > > > > > > > already
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> > for
> > > > > > catalog
> > > > > > > > than
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> > > > should
> > > > > be
> > > > > > > > lazy
> > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > instead of asynchronous registration. I
> > have
> > > > > > already
> > > > > > > > > > updated
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon
> FY <
> > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> > > > method
> > > > > in
> > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > confuse users whether a method named
> > > > > `addCatalog`
> > > > > > > or
> > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > And as you mentioned, the memory
> catalog
> > > is a
> > > > > > > > `cache`,
> > > > > > > > > I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > of `cache` should not be exposed to
> > users.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1]
> for
> > > > this
> > > > > > > FLIP.
> > > > > > > > > > Please
> > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > > > > discussion
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> > > > implement
> > > > > it
> > > > > > > for
> > > > > > > > > > > > different
> > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think there is a gap between the
> > current
> > > > FLIP
> > > > > > > > design
> > > > > > > > > > and
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > What about the proposal of the
> discussion
> > > in
> > > > > > thread
> > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > >
> > > > > > > > >
> > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> > with
> > > > the
> > > > > > > same
> > > > > > > > > name
> > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > complex to me. I think it'll be
> better
> > to
> > > > > > explain
> > > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > these methods and give a brief
> example
> > in
> > > > > this
> > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > > still
> > > > > > looks
> > > > > > > > > > confused
> > > > > > > > > > > > to
> > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > > CatalogStore,
> > > > > > > > > how
> > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > > opinion,
> > > > > > > you
> > > > > > > > > can
> > > > > > > > > > > > decide
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement,
> which
> > > > method
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`?
> I'm
> > > > > > wondering
> > > > > > > > > > whether
> > > > > > > > > > > > user
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> provide a
> > > > > default
> > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 4: About asynchronous registration
> for
> > > > > catalog.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > > > > CATALOG,
> > > > > > > the
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration method is used by
> default.
> > > > > > > > > > > > > > > > > > > > If asynchronous registration is the
> > > default
> > > > > > > > behavior,
> > > > > > > > > > it
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > > > > registration
> > > > > > > > just
> > > > > > > > > > > like
> > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > > > `registerCatalog`
> > > > > > > > > be
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous registration,
> it
> > > may
> > > > > well
> > > > > > > > that
> > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > executes successfully, but then the
> > > > following
> > > > > > > > CREATE
> > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > > > > I think it's a break change which
> > should
> > > be
> > > > > > > > > highlighted
> > > > > > > > > > > in
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > registration, I
> > > > > > would
> > > > > > > > > like
> > > > > > > > > > to
> > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > will be an executor to open or
> register
> > > > > catalog
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > initialized
> > > > if
> > > > > it
> > > > > > > has
> > > > > > > > > not
> > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > yet. If the initialization process
> > fails,
> > > > > these
> > > > > > > > > > > statements
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > It looks more like lazy
> initialization
> > > for
> > > > > > > catalog
> > > > > > > > > than
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> jinfeng1995@gmail.com
> > >
> > > > > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > > > > persistence
> > > > > > > of
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> > > > interest
> > > > > in
> > > > > > > > this
> > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > How to handle a catalog with the
> same
> > > > name
> > > > > > that
> > > > > > > > > > exists
> > > > > > > > > > > > for
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I believe this is a crucial point.
> > Based
> > > on
> > > > > my
> > > > > > > > > current
> > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > understanding, the Map<String,
> Catalog>
> > > > > > catalogs
> > > > > > > > will
> > > > > > > > > > > serve
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > for instantiated catalogs and have
> the
> > > > > highest
> > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There are three methods that can
> affect
> > > the
> > > > > > > > > Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This method puts the catalog instance
> > > into
> > > > > the
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > catalogName)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This method removes the catalog
> > instance
> > > > > > > > > corresponding
> > > > > > > > > > to
> > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> catalogs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > > corresponding
> > > > > > > > catalog
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If
> > the
> > > > > > catalog
> > > > > > > > > does
> > > > > > > > > > > not
> > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > configuration
> > > > > from
> > > > > > > the
> > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > initialized
> > > > > > Catalog
> > > > > > > > > > instance
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The following two methods only modify
> > the
> > > > > > > > > configuration
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > > > Map<String,
> > > > > > > > String>
> > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This method saves the properties to
> the
> > > > > > > > catalogStore
> > > > > > > > > > and
> > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > whether there is a catalogName with
> the
> > > > same
> > > > > > > name.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > > This method removes the specified
> > > > > configuration
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The following are possible conflict
> > > > > scenarios:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> > > > already
> > > > > > > > exists
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> Map<String,
> > > > > > Catalog>,
> > > > > > > > the
> > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > method
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > > > > Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> > > > already
> > > > > > > > exists
> > > > > > > > > in
> > > > > > > > > > > > both
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> > > Catalog>,
> > > > > the
> > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method
> > will
> > > > > fail.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> > > > already
> > > > > > > > exists
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > addCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > String> properties) method can
> directly
> > > > save
> > > > > > the
> > > > > > > > > > > properties
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > catalogStore, but the
> getCatalog(String
> > > > > > > > catalogName)
> > > > > > > > > > > method
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > use the new properties for
> > initialization
> > > > > > because
> > > > > > > > the
> > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > catalog instance already exists in
> > > catalogs
> > > > > and
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > unregisterCatalog(String
> > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > remove the instance corresponding to
> > the
> > > > > > original
> > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > > > > `Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > > > > Map<String,
> > > > > > > > > > String>
> > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This could potentially lead to
> > > confusion. I
> > > > > > > suggest
> > > > > > > > > > > > changing
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > addCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > String> properties), as previously
> > > > mentioned
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > catalogName,Catalog
> > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> instance,
> > > > > adding
> > > > > > > the
> > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > parameter to the
> registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > method may not necessarily result in
> > lazy
> > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Do we need to think about
> encryption
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think encryption is necessary, but
> > > > perhaps
> > > > > > the
> > > > > > > > > > > encryption
> > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > should be implemented in the
> > CatalogStore
> > > > > when
> > > > > > > the
> > > > > > > > > data
> > > > > > > > > > > is
> > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > FileCatalogStore
> > > > > could
> > > > > > > > > encrypt
> > > > > > > > > > > the
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > > > MemoryCatalogStore
> > > > > > > > > > does
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > > > MemoryCatalogStore
> > > > > > > > > as
> > > > > > > > > > > the
> > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > > > > configurations
> > > > > > > > in
> > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > > > > asynchronous
> > > > > > > > > loading
> > > > > > > > > > > of
> > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > would need to introduce additional
> > cache.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > What if asynchronous registration
> > > failed?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This is also a critical concern.
> When
> > > > > > executing
> > > > > > > > DDL,
> > > > > > > > > > > DQL,
> > > > > > > > > > > > or
> > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > statements that reference a specific
> > > > Catalog,
> > > > > > the
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > will be initialized if it has not
> been
> > > > > > > initialized
> > > > > > > > > yet.
> > > > > > > > > > > If
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > initialization process fails, these
> > > > > statements
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > be
> > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog>
> catalogs
> > > and
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the CatalogManager all have the same
> > > > catalog
> > > > > > > name,
> > > > > > > > > but
> > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> referring
> > > to
> > > > > the
> > > > > > > > > previous
> > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > key principle is that the Catalog
> that
> > > has
> > > > > been
> > > > > > > > most
> > > > > > > > > > > > recently
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > will be given priority for subsequent
> > > use.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM
> Xianxun
> > > Ye
> > > > <
> > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> > > > believe
> > > > > > this
> > > > > > > > > Flip
> > > > > > > > > > > > could
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > for users. The firm I work for also
> > > > manages a
> > > > > > lot
> > > > > > > > of
> > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > submission of tasks becomes slow
> > because
> > > of
> > > > > > > > loading a
> > > > > > > > > > > > number
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the user's
> > > Flink
> > > > > SQL
> > > > > > > > > through
> > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs to
> > improve
> > > > the
> > > > > > > speed
> > > > > > > > of
> > > > > > > > > > > task
> > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > > > > questions:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog with
> > CREATE
> > > > > > CATALOG,
> > > > > > > > the
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration method is used by
> default.
> > > > What
> > > > > if
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > asynchronous
> > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. I also have the same question
> with
> > > > > > Shammon’s
> > > > > > > > > first
> > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > > > CatalogStore
> > > > > > > > > > > catalogstore
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogManager all have the same
> > catalog
> > > > > name,
> > > > > > > but
> > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > different catalog instances. For
> > example:
> > > > > > > > catalog1->
> > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog
> > in
> > > > the
> > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> > `catalogs`
> > > > and
> > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jark Wu <im...@gmail.com>.
Hi Feng,
This is a useful FLIP. Thanks for starting this discussion.
The current design looks pretty good to me. I just have some minor comments.
1. How to register the CatalogStore for Table API? I think the CatalogStore
should be immutable once TableEnv is created. Otherwise, there might be
some data inconsistencies when CatalogStore is changed.
2. Why does the CatalogStoreFactory interface only have a default method,
not an interface method?
3. Please mention the alternative API in Javadoc for the deprecated
`registerCatalog`.
4. In the "Compatibility" section, would be better to mention the changed
behavior of CREATE CATALOG statement if FileCatalogStore (or other
persisted catalog store) is used.
Best,
Jark
On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
> Hi ， thanks all for reviewing the flip.
>
> @Ron
>
> > Regarding the CatalogStoreFactory#createCatalogStore method, do we need
> to provide a default implementation?
>
> Yes, we will provide a default InMemoryCatalogStoreFactory to create an
> InMemoryCatalogStore.
>
> > If we get a Catalog from CatalogStore, after initializing it, whether we
> put it in Map<String, Catalog> catalogs again?
>
> Yes, in the current design, catalogs are stored as snapshots, and once
> initialized, the Catalog will be placed in the Map<String, Catalog>
> catalogs.
> Subsequently, the Map<String, Catalog> catalogs will be the primary source
> for obtaining the corresponding Catalog.
>
> > how about renaming them to `catalog.store.type` and
> `catalog.store.path`?
>
> I think it is okay. Adding "sql" at the beginning may seem a bit strange. I
> will update the FLIP.
>
>
>
> @Shammon
>
> Thank you for the review. I have made the necessary corrections.
> Regarding the modifications made to the Public Interface, I have also
> included the relevant changes to the `TableEnvironment`.
>
>
> Best,
> Feng
>
>
> On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com> wrote:
>
> > Hi feng,
> >
> > Thanks for updating, I have some minor comments
> >
> > 1. The modification of `CatalogManager` should not be in `Public
> > Interfaces`, it is not a public interface.
> >
> > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > `CatalogStoreFactory`
> >
> > 3. The code `Optional<Catalog> optionalDescriptor =
> > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > `Optional<CatalogDescriptor> optionalDescriptor =
> > catalogStore.get(catalogName);`
> >
> > Best,
> > Shammon FY
> >
> >
> > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
> >
> > > Hi, Feng
> > >
> > > Thanks for driving this FLIP, this proposal is very useful for catalog
> > > management.
> > > I have some small questions:
> > >
> > > 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > need
> > > to provide a default implementation?
> > > 2. If we get Catalog from CatalogStore, after initializing it, whether
> we
> > > put it to Map<String, Catalog> catalogs again?
> > > 3. Regarding the options `sql.catalog.store.type` and
> > > `sql.catalog.store.file.path`, how about renaming them to
> > > `catalog.store.type` and `catalog.store.path`?
> > >
> > > Best,
> > > Ron
> > >
> > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > >
> > > > Hi yuxia
> > > >
> > > > > But from the code in Proposed Changes, once we register the
> Catalog,
> > > we
> > > > initialize it and open it. right?
> > > >
> > > > Yes, In order to avoid inconsistent semantics of the original CREATE
> > > > CATALOG DDL, Catalog will be directly initialized in registerCatalog
> so
> > > > that parameter validation can be performed.
> > > >
> > > > In the current design, lazy initialization is mainly reflected in
> > > > getCatalog. If CatalogStore has already saved some catalog
> > > configurations,
> > > > only initialization is required in getCatalog.
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Mon, May 29, 2023 at 8:27 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > > wrote:
> > > >
> > > > > Hi, Feng.
> > > > > I'm trying to understanding the meaning of *lazy initialization*.
> If
> > > i'm
> > > > > wrong, please correct me.
> > > > >
> > > > > IIUC, lazy initialization means only you need to access the
> catalog,
> > > then
> > > > > you initialize it. But from the code in Proposed Changes, once we
> > > > register
> > > > > the Catalog,
> > > > > we initialize it and open it. right?
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > > ----- 原始邮件 -----
> > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > configuration
> > > > > and asynchronous registration
> > > > >
> > > > > Hi Feng,
> > > > >
> > > > > Thanks for your effort! +1 for the proposal.
> > > > >
> > > > > One of the major changes is that current design will provide
> > > > > Map<String,Catalog> catalogs as a snapshot instead of a cache,
> which
> > > > means
> > > > > once it has been initialized, any changes done by other sessions
> will
> > > not
> > > > > affect it. Point 6 described follow-up options for further
> > improvement.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi all, I would like to update you on the latest progress of the
> > > FLIP.
> > > > > >
> > > > > >
> > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai
> Fang
> > > > and I
> > > > > > had an offline discussion regarding the overall solution for
> Flink
> > > > > > CatalogStore. We have reached a consensus and I have updated the
> > > final
> > > > > > solution in FLIP.
> > > > > >
> > > > > > Next, let me briefly describe the entire design:
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Introduce CatalogDescriptor to store catalog configuration
> > similar
> > > > to
> > > > > > TableDescriptor.
> > > > > > 2.
> > > > > >
> > > > > > The two key functions of CatalogStore - void
> storeCatalog(String
> > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > getCatalog(String)
> > > > > > will both use CatalogDescriptor instead of Catalog instance.
> > This
> > > > way,
> > > > > > CatalogStore will only be responsible for saving and
> retrieving
> > > > > catalog
> > > > > > configurations without having to initialize catalogs.
> > > > > > 3.
> > > > > >
> > > > > > The default registerCatalog(String catalogName, Catalog
> catalog)
> > > > > > function in CatalogManager will be marked as deprecated.
> > > > > > 4.
> > > > > >
> > > > > > A new function registerCatalog(String catalogName,
> > > CatalogDescriptor
> > > > > > catalog) will be added to serve as the default registration
> > > function
> > > > > for
> > > > > > catalogs in CatalogManager.
> > > > > > 5.
> > > > > >
> > > > > > Map<String,Catalog> catalogs in CataloManager will remain
> > > unchanged
> > > > > and
> > > > > > save initialized catalogs.This means that deletion operations
> > from
> > > > one
> > > > > > session won't synchronize with other sessions.
> > > > > > 6.
> > > > > >
> > > > > > To support multi-session synchronization scenarios for
> deletions
> > > > later
> > > > > > on we should make Map<String,Catalog>catalogs
> configurable.There
> > > may
> > > > > be
> > > > > > three possible situations:
> > > > > >
> > > > > > a.Default caching of all initialized catalogs
> > > > > >
> > > > > > b.Introduction of LRU cache logic which can automatically
> clear
> > > > > > long-unused catalogs.
> > > > > >
> > > > > > c.No caching of any instances; each call to getCatalog
> creates a
> > > new
> > > > > > instance.
> > > > > >
> > > > > >
> > > > > > This is the document for discussion:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > >
> > > > > > This is the final proposal document:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > >
> > > > > >
> > > > > > Thank you very much for your attention and suggestions on this
> > > FLIP. A
> > > > > > special thanks to Hang Ruan for his suggestions on the entire
> > design
> > > > and
> > > > > > organizing offline discussions.
> > > > > >
> > > > > > If you have any further suggestions or feedback about this FLIP
> > > please
> > > > > feel
> > > > > > free to share.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Feng
> > > > > >
> > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > Thanks for improving the FLIP. It looks good to me. We could
> > still
> > > > > > > reconsider in the future how to provide more common built-in
> > cache
> > > > > > > functionality in CatalogManager so that not every CatalogSotre
> > > > > > > implementation has to take care of it.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <jinfeng1995@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Jing,
> > > > > > > >
> > > > > > > > Thanks for your reply.
> > > > > > > >
> > > > > > > > > There might be more such issues. I would suggest you
> > > completely
> > > > > walk
> > > > > > > > through the FLIP again and fix those issues
> > > > > > > >
> > > > > > > > I am very sorry for my carelessness and at the same time, I
> > > greatly
> > > > > > > > appreciate your careful review. I have thoroughly checked the
> > > > entire
> > > > > > FLIP
> > > > > > > > and made corrections to these issues.
> > > > > > > >
> > > > > > > >
> > > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > > CatalogManager
> > > > > > > > could work without Optional CatalogStore being configured.
> > > > > > > >
> > > > > > > > Yes, in the original design, CatalogStore was not necessary
> > > because
> > > > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> > > catalog
> > > > > > > > instances.
> > > > > > > > However, this caused inconsistency issues. Therefore, I
> > modified
> > > > this
> > > > > > > part
> > > > > > > > of the design and removed Map<String, Catalog> catalogs from
> > > > > > > > CatalogManager.
> > > > > > > > At the same time, InMemoryCatalog will serve as the default
> > > > > > CatalogStore
> > > > > > > > to save catalogs in memory and replace the functionality of
> > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > The previous plan that kept Map<String,Catalog>catalogs has
> > been
> > > > > moved
> > > > > > to
> > > > > > > > Rejected Alternatives.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng
> > > > > > > >
> > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > There are still many places contain inconsistent content,
> > e.g.
> > > > > > > > >
> > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > 2. The java comment of the method registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog catalog) in CatalogManager does not tell what the
> > > method
> > > > is
> > > > > > > > doing.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > There might be more such issues. I would suggest you
> > completely
> > > > > walk
> > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > >
> > > > > > > > > About the inMemoryCatalogStore, do you mean that you will
> > build
> > > > the
> > > > > > > cache
> > > > > > > > > functionality in the CatalogStore? This is a very different
> > > > design
> > > > > > > > concept
> > > > > > > > > from what the current FLIP described. If I am not mistaken,
> > > with
> > > > > the
> > > > > > > > > current FLIP design, CatalogManager could work without
> > Optional
> > > > > > > > > CatalogStore being configured. That is the reason why I
> > > mentioned
> > > > > in
> > > > > > > the
> > > > > > > > > last email that the example code wrt the Optional
> > CatalogStore
> > > is
> > > > > not
> > > > > > > > > correct.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > jinfeng1995@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jing
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > >
> > > > > > > > > > Thank you very much for the careful review. I have
> already
> > > made
> > > > > the
> > > > > > > > > > relevant changes.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > > scenarios, I
> > > > > > > > am
> > > > > > > > > > not
> > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > >
> > > > > > > > > > Currently, there are conflicts in multiple scenarios with
> > the
> > > > > > current
> > > > > > > > > > design. I am thinking whether we should remove
> 'Map<String,
> > > > > > Catalog>'
> > > > > > > > and
> > > > > > > > > > make Cache the default behavior of InMemoryCatalogStore.
> > This
> > > > > way,
> > > > > > > > users
> > > > > > > > > > can implement their own CatalogStore to achieve
> > > multi-instance
> > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for working on the FLIP. There are still some
> NIT
> > > > issues
> > > > > > in
> > > > > > > > the
> > > > > > > > > > FLIP
> > > > > > > > > > > like:
> > > > > > > > > > >
> > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > > > > > > CatalogStore
> > > > > > > > > > > instead of Optional in the code example. It should be
> > fine
> > > to
> > > > > use
> > > > > > > it
> > > > > > > > as
> > > > > > > > > > > pseudo code for now and update it after you submit the
> > PR.
> > > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> > rejected
> > > > > > section
> > > > > > > > > which
> > > > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > > > >
> > > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > scenarios,
> > > > > > I
> > > > > > > am
> > > > > > > > > not
> > > > > > > > > > > sure if this is the intended behaviour. If Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > is
> > > > > > > > > > > used as a cache, it should be invalid, once the related
> > > > catalog
> > > > > > has
> > > > > > > > > been
> > > > > > > > > > > removed from the CatalogStore by another instance. Did
> I
> > > miss
> > > > > > > > > something?
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > jinfeng1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > >
> > > > > > > > > > > > @Jing
> > > > > > > > > > > >
> > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > I think this is a good function name, I have updated
> it
> > > in
> > > > > the
> > > > > > > > > > > > documentation.
> > > > > > > > > > > >
> > > > > > > > > > > > > Some common cache features should be implemented
> > > > > > > > > > > > Thank you for the suggestion. If alternative 1 turns
> > out
> > > to
> > > > > be
> > > > > > > more
> > > > > > > > > > > > appropriate later, I will improve this part of the
> > > content.
> > > > > > > > > > > >
> > > > > > > > > > > > > As the above discussion moves forward, the option 2
> > > > > solution
> > > > > > > > looks
> > > > > > > > > > more
> > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > Yes, after discussing with Shammon offline, we think
> > that
> > > > > > > solution
> > > > > > > > 2
> > > > > > > > > > > might
> > > > > > > > > > > > be more suitable and also avoid any inconsistency
> > issues.
> > > > > > > > > > > >
> > > > > > > > > > > > > There are some inconsistent descriptions in the
> > > content.
> > > > > > Would
> > > > > > > > you
> > > > > > > > > > > like
> > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > I will do my best to improve the document and
> > appreciate
> > > > your
> > > > > > > > > > > suggestions.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > can you put the unselected option in `Rejected
> > > > > Alternatives`
> > > > > > > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > zjureel@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I found there are two options in `Proposed
> Changes`,
> > > can
> > > > > you
> > > > > > > put
> > > > > > > > > the
> > > > > > > > > > > > > unselected option in `Rejected Alternatives`? I
> think
> > > > this
> > > > > > may
> > > > > > > > help
> > > > > > > > > > us
> > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for raising this FLIP. I am still confused
> > > after
> > > > > > > > > completely
> > > > > > > > > > > > > reading
> > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > > addCatalog()
> > > > > > have
> > > > > > > > no
> > > > > > > > > > big
> > > > > > > > > > > > > > difference based on their names. One of them is
> > > > > responsible
> > > > > > > for
> > > > > > > > > > data
> > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> > > catalogs
> > > > is
> > > > > > > used
> > > > > > > > > as a
> > > > > > > > > > > > cache
> > > > > > > > > > > > > > and catalogStore is used for data persistence. I
> > > would
> > > > > > > suggest
> > > > > > > > > > > > describing
> > > > > > > > > > > > > > their purpose conceptually and clearly in the
> FLIP.
> > > > Some
> > > > > > > common
> > > > > > > > > > cache
> > > > > > > > > > > > > > features should be implemented, i.e. data in the
> > > cache
> > > > > and
> > > > > > > the
> > > > > > > > > > store
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > be consistent. Same Catalog instance should be
> > found
> > > in
> > > > > the
> > > > > > > > store
> > > > > > > > > > and
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the cache(either it has been initialized or it
> will
> > > be
> > > > > lazy
> > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > for the same catalog name. The consistency will
> be
> > > > taken
> > > > > > care
> > > > > > > > of
> > > > > > > > > > > while
> > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > 3. As the above discussion moves forward, the
> > option
> > > 2
> > > > > > > solution
> > > > > > > > > > looks
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> > > issues
> > > > > > > > mentioned
> > > > > > > > > > > > > > previously with option 1 are not solved yet. Do
> you
> > > > still
> > > > > > > want
> > > > > > > > to
> > > > > > > > > > > > propose
> > > > > > > > > > > > > > both options and ask for suggestions for both of
> > > them?
> > > > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > > > inconsistent
> > > > > > > > > > > descriptions
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the content. Would you like to clean them up?
> > > Thanks!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > jinfeng1995@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you for your response, and I completely
> > agree
> > > > > with
> > > > > > > your
> > > > > > > > > > point
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > Initially, I may have over complicated the
> whole
> > > > issue.
> > > > > > > First
> > > > > > > > > and
> > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > we need to consider the persistence of the
> > > Catalog's
> > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > If we only need to provide persistence for
> > Catalog
> > > > > > > > > Configuration,
> > > > > > > > > > > we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > > > interface.
> > > > > > > > > > > > > > > This method can convert a Catalog instance to a
> > > > > > Map<String,
> > > > > > > > > > String>
> > > > > > > > > > > > > > > properties, and the default implementation will
> > > throw
> > > > > an
> > > > > > > > > > exception.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > * Returns a map containing the properties of
> > the
> > > > > > catalog
> > > > > > > > > > object.
> > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > * @return a map containing the properties of
> > the
> > > > > > catalog
> > > > > > > > > > object
> > > > > > > > > > > > > > > * @throws UnsupportedOperationException if
> the
> > > > > > > > implementing
> > > > > > > > > > > class
> > > > > > > > > > > > > does
> > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > * the default implementation of this method
> > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > > > > > > throw new
> > UnsupportedOperationException("Please
> > > > > > > implement
> > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. If the user has configured a CatalogStore,
> > the
> > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > will be called when registering a Catalog
> > instance
> > > > with
> > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > catalog).
> > > > > The
> > > > > > > > > Catalog
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > will be converted to a Map<String, String>
> > > properties
> > > > > and
> > > > > > > > saved
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore. If some Catalog instances do not
> > > > > implement
> > > > > > > this
> > > > > > > > > > > method,
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. If the user does not use a CatalogStore,
> the
> > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > > will not be called, ensuring consistency with
> the
> > > > > > original
> > > > > > > > > > process.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> catalogs
> > > and
> > > > > the
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > at
> > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > we can start from the Catalog itself and
> provide
> > a
> > > > > > > dedicated
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > public class ConfigurationCatalog implements
> > > Catalog
> > > > {
> > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > > properties)
> > > > {
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have no questions about the functionality
> of
> > > > > > > > > `CatalogStore`,
> > > > > > > > > > > but
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > different operations of multiple
> > > `registerCatalog`
> > > > or
> > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > `CatalogManager`. The implementation in Trino
> > is
> > > > also
> > > > > > the
> > > > > > > > > same,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > > `createCatalog`,
> > > > > > > > which
> > > > > > > > > > > will
> > > > > > > > > > > > > save
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > `registerCatalog`
> > > > > > or
> > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> > catalogs`
> > > > or
> > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > catalogStore`. As you mentioned before, the
> > > > > > `Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How
> > > about
> > > > > > > saving
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > `CatalogStore
> > > > > > > > > catalogStore`
> > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > when it is registered or added in
> > > `CatalogManager`?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you for your reply. I am very sorry
> for
> > > the
> > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > by my deviation from the original
> discussion.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> > > this
> > > > > FLIP
> > > > > > > > > > > > > > > > > Yes, there was indeed such a discussion
> > before.
> > > > > > > However,
> > > > > > > > > in
> > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > > > CatalogManager
> > > > > > > > > > itself
> > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > need to change much. *We cannot only
> persist
> > > > > Catalog
> > > > > > > > > > instances
> > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > not be enough to save Catalogs, because in
> > the
> > > > end
> > > > > we
> > > > > > > > still
> > > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > the configurations corresponding to the
> > Catalog
> > > > > > > > instances.
> > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> > interface
> > > > for
> > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > persistence. Regarding this part, I also
> took
> > > > > > > inspiration
> > > > > > > > > > from
> > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with
> > the
> > > > > same
> > > > > > > name
> > > > > > > > > > > looks a
> > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > Thank you for the suggestion. I will
> provide
> > a
> > > > > > detailed
> > > > > > > > > > > > description
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > code examples for this part, and add it to
> > the
> > > > > FLIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> still
> > > > looks
> > > > > > > > confused
> > > > > > > > > > to
> > > > > > > > > > > > me.
> > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > CatalogStore,
> > > > > > how
> > > > > > > > > about
> > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > opinion,
> > > > you
> > > > > > can
> > > > > > > > > decide
> > > > > > > > > > > to
> > > > > > > > > > > > > take
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me. I
> > > don't
> > > > > see
> > > > > > > any
> > > > > > > > > > > problem
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which
> > method
> > > > will
> > > > > > be
> > > > > > > > > > called?
> > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > wondering
> > > > > > > whether
> > > > > > > > > user
> > > > > > > > > > > can
> > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > For CREATE CATALOG, my original design was
> to
> > > add
> > > > > it
> > > > > > > > > directly
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> > > existing
> > > > > > > logic.
> > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > we can do both: save the configuration to
> the
> > > > > > > > CatalogStore
> > > > > > > > > > and
> > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> > > default
> > > > > > > > > > implmentation
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > I think it is necessary, otherwise we would
> > > need
> > > > to
> > > > > > > > > introduce
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > Map to store the configuration for lazy
> > > loading.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > catalog.
> > > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > > catalogName,
> > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > made into an asynchronous interface because
> > > > Catalog
> > > > > > is
> > > > > > > > > > already
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> for
> > > > > catalog
> > > > > > > than
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> > > should
> > > > be
> > > > > > > lazy
> > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > instead of asynchronous registration. I
> have
> > > > > already
> > > > > > > > > updated
> > > > > > > > > > > the
> > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > > > > > > zjureel@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> > > method
> > > > in
> > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > confuse users whether a method named
> > > > `addCatalog`
> > > > > > or
> > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > And as you mentioned, the memory catalog
> > is a
> > > > > > > `cache`,
> > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > of `cache` should not be exposed to
> users.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> > > this
> > > > > > FLIP.
> > > > > > > > > Please
> > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > > > discussion
> > > > > > is
> > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> > > implement
> > > > it
> > > > > > for
> > > > > > > > > > > different
> > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think there is a gap between the
> current
> > > FLIP
> > > > > > > design
> > > > > > > > > and
> > > > > > > > > > > that
> > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > What about the proposal of the discussion
> > in
> > > > > thread
> > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > >
> > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> with
> > > the
> > > > > > same
> > > > > > > > name
> > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > complex to me. I think it'll be better
> to
> > > > > explain
> > > > > > > it
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > java
> > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > these methods and give a brief example
> in
> > > > this
> > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > still
> > > > > looks
> > > > > > > > > confused
> > > > > > > > > > > to
> > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > CatalogStore,
> > > > > > > > how
> > > > > > > > > > > about
> > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > opinion,
> > > > > > you
> > > > > > > > can
> > > > > > > > > > > decide
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which
> > > method
> > > > > > will
> > > > > > > be
> > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > > > wondering
> > > > > > > > > whether
> > > > > > > > > > > user
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to provide a
> > > > default
> > > > > > > > > > > implmentation
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > > catalog.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > > > CATALOG,
> > > > > > the
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > > > > > > If asynchronous registration is the
> > default
> > > > > > > behavior,
> > > > > > > > > it
> > > > > > > > > > > > there
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > > > registration
> > > > > > > just
> > > > > > > > > > like
> > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > > `registerCatalog`
> > > > > > > > be
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > IIUC, in asynchronous registration, it
> > may
> > > > well
> > > > > > > that
> > > > > > > > > > CREATE
> > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > executes successfully, but then the
> > > following
> > > > > > > CREATE
> > > > > > > > > > TABLE
> > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > > > I think it's a break change which
> should
> > be
> > > > > > > > highlighted
> > > > > > > > > > in
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > registration, I
> > > > > would
> > > > > > > > like
> > > > > > > > > to
> > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > will be an executor to open or register
> > > > catalog
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> initialized
> > > if
> > > > it
> > > > > > has
> > > > > > > > not
> > > > > > > > > > > been
> > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > yet. If the initialization process
> fails,
> > > > these
> > > > > > > > > > statements
> > > > > > > > > > > > will
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> > for
> > > > > > catalog
> > > > > > > > than
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <jinfeng1995@gmail.com
> >
> > > > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > > > persistence
> > > > > > of
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> > > interest
> > > > in
> > > > > > > this
> > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > How to handle a catalog with the same
> > > name
> > > > > that
> > > > > > > > > exists
> > > > > > > > > > > for
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I believe this is a crucial point.
> Based
> > on
> > > > my
> > > > > > > > current
> > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > understanding, the Map<String, Catalog>
> > > > > catalogs
> > > > > > > will
> > > > > > > > > > serve
> > > > > > > > > > > > as
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > for instantiated catalogs and have the
> > > > highest
> > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There are three methods that can affect
> > the
> > > > > > > > Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This method puts the catalog instance
> > into
> > > > the
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> catalogName)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This method removes the catalog
> instance
> > > > > > > > corresponding
> > > > > > > > > to
> > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > corresponding
> > > > > > > catalog
> > > > > > > > > > > > instance
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If
> the
> > > > > catalog
> > > > > > > > does
> > > > > > > > > > not
> > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > retrieves the corresponding
> configuration
> > > > from
> > > > > > the
> > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > initializes it, and puts the
> initialized
> > > > > Catalog
> > > > > > > > > instance
> > > > > > > > > > > > into
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The following two methods only modify
> the
> > > > > > > > configuration
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > > Map<String,
> > > > > > > String>
> > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This method saves the properties to the
> > > > > > > catalogStore
> > > > > > > > > and
> > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > whether there is a catalogName with the
> > > same
> > > > > > name.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > This method removes the specified
> > > > configuration
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The following are possible conflict
> > > > scenarios:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> > > already
> > > > > > > exists
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore but not in the Map<String,
> > > > > Catalog>,
> > > > > > > the
> > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > Catalog
> > > > > > > catalog)
> > > > > > > > > > method
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > > > Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> > > already
> > > > > > > exists
> > > > > > > > in
> > > > > > > > > > > both
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> > Catalog>,
> > > > the
> > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method
> will
> > > > fail.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> > > already
> > > > > > > exists
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > addCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > String> properties) method can directly
> > > save
> > > > > the
> > > > > > > > > > properties
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > > > > > > catalogName)
> > > > > > > > > > method
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > use the new properties for
> initialization
> > > > > because
> > > > > > > the
> > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > catalog instance already exists in
> > catalogs
> > > > and
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > Therefore, using the
> > > unregisterCatalog(String
> > > > > > > > > > catalogName)
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > remove the instance corresponding to
> the
> > > > > original
> > > > > > > > > > > catalogName
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > > > `Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > > > Map<String,
> > > > > > > > > String>
> > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This could potentially lead to
> > confusion. I
> > > > > > suggest
> > > > > > > > > > > changing
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > method name, perhaps to
> addCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > String> properties), as previously
> > > mentioned
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > catalogName,Catalog
> > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Since a catalog is already an instance,
> > > > adding
> > > > > > the
> > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > parameter to the registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > method may not necessarily result in
> lazy
> > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think encryption is necessary, but
> > > perhaps
> > > > > the
> > > > > > > > > > encryption
> > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > should be implemented in the
> CatalogStore
> > > > when
> > > > > > the
> > > > > > > > data
> > > > > > > > > > is
> > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > saved. For instance, the
> FileCatalogStore
> > > > could
> > > > > > > > encrypt
> > > > > > > > > > the
> > > > > > > > > > > > > data
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > > MemoryCatalogStore
> > > > > > > > > does
> > > > > > > > > > > not
> > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > > MemoryCatalogStore
> > > > > > > > as
> > > > > > > > > > the
> > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > > > configurations
> > > > > > > in
> > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > > > asynchronous
> > > > > > > > loading
> > > > > > > > > > of
> > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > would need to introduce additional
> cache.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > What if asynchronous registration
> > failed?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This is also a critical concern. When
> > > > > executing
> > > > > > > DDL,
> > > > > > > > > > DQL,
> > > > > > > > > > > or
> > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > statements that reference a specific
> > > Catalog,
> > > > > the
> > > > > > > > > Catalog
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > will be initialized if it has not been
> > > > > > initialized
> > > > > > > > yet.
> > > > > > > > > > If
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > initialization process fails, these
> > > > statements
> > > > > > will
> > > > > > > > not
> > > > > > > > > > be
> > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs
> > and
> > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the CatalogManager all have the same
> > > catalog
> > > > > > name,
> > > > > > > > but
> > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This issue can be resolved by referring
> > to
> > > > the
> > > > > > > > previous
> > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > key principle is that the Catalog that
> > has
> > > > been
> > > > > > > most
> > > > > > > > > > > recently
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > will be given priority for subsequent
> > use.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun
> > Ye
> > > <
> > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> > > believe
> > > > > this
> > > > > > > > Flip
> > > > > > > > > > > could
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > for users. The firm I work for also
> > > manages a
> > > > > lot
> > > > > > > of
> > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > submission of tasks becomes slow
> because
> > of
> > > > > > > loading a
> > > > > > > > > > > number
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > We obtain the catalogs in the user's
> > Flink
> > > > SQL
> > > > > > > > through
> > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > to avoid loading all catalogs to
> improve
> > > the
> > > > > > speed
> > > > > > > of
> > > > > > > > > > task
> > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > > > questions:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. When creating a catalog with
> CREATE
> > > > > CATALOG,
> > > > > > > the
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration method is used by default.
> > > What
> > > > if
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > failed? And how to implement
> asynchronous
> > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. I also have the same question with
> > > > > Shammon’s
> > > > > > > > first
> > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > > CatalogStore
> > > > > > > > > > catalogstore
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogManager all have the same
> catalog
> > > > name,
> > > > > > but
> > > > > > > > > > > > correponding
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > different catalog instances. For
> example:
> > > > > > > catalog1->
> > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog
> in
> > > the
> > > > > > > > > > catalogstore.
> > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> `catalogs`
> > > and
> > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > >
> > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi ， thanks all for reviewing the flip.
@Ron
> Regarding the CatalogStoreFactory#createCatalogStore method, do we need
to provide a default implementation?
Yes, we will provide a default InMemoryCatalogStoreFactory to create an
InMemoryCatalogStore.
> If we get a Catalog from CatalogStore, after initializing it, whether we
put it in Map<String, Catalog> catalogs again?
Yes, in the current design, catalogs are stored as snapshots, and once
initialized, the Catalog will be placed in the Map<String, Catalog>
catalogs.
Subsequently, the Map<String, Catalog> catalogs will be the primary source
for obtaining the corresponding Catalog.
> how about renaming them to `catalog.store.type` and
`catalog.store.path`?
I think it is okay. Adding "sql" at the beginning may seem a bit strange. I
will update the FLIP.
@Shammon
Thank you for the review. I have made the necessary corrections.
Regarding the modifications made to the Public Interface, I have also
included the relevant changes to the `TableEnvironment`.
Best,
Feng
On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com> wrote:
> Hi feng,
>
> Thanks for updating, I have some minor comments
>
> 1. The modification of `CatalogManager` should not be in `Public
> Interfaces`, it is not a public interface.
>
> 2. `@PublicEvolving` should be added for `CatalogStore` and
> `CatalogStoreFactory`
>
> 3. The code `Optional<Catalog> optionalDescriptor =
> catalogStore.get(catalogName);` in the `CatalogManager` should be
> `Optional<CatalogDescriptor> optionalDescriptor =
> catalogStore.get(catalogName);`
>
> Best,
> Shammon FY
>
>
> On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
>
> > Hi, Feng
> >
> > Thanks for driving this FLIP, this proposal is very useful for catalog
> > management.
> > I have some small questions:
> >
> > 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we
> need
> > to provide a default implementation?
> > 2. If we get Catalog from CatalogStore, after initializing it, whether we
> > put it to Map<String, Catalog> catalogs again?
> > 3. Regarding the options `sql.catalog.store.type` and
> > `sql.catalog.store.file.path`, how about renaming them to
> > `catalog.store.type` and `catalog.store.path`?
> >
> > Best,
> > Ron
> >
> > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> >
> > > Hi yuxia
> > >
> > > > But from the code in Proposed Changes, once we register the Catalog,
> > we
> > > initialize it and open it. right?
> > >
> > > Yes, In order to avoid inconsistent semantics of the original CREATE
> > > CATALOG DDL, Catalog will be directly initialized in registerCatalog so
> > > that parameter validation can be performed.
> > >
> > > In the current design, lazy initialization is mainly reflected in
> > > getCatalog. If CatalogStore has already saved some catalog
> > configurations,
> > > only initialization is required in getCatalog.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Mon, May 29, 2023 at 8:27 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > wrote:
> > >
> > > > Hi, Feng.
> > > > I'm trying to understanding the meaning of *lazy initialization*. If
> > i'm
> > > > wrong, please correct me.
> > > >
> > > > IIUC, lazy initialization means only you need to access the catalog,
> > then
> > > > you initialize it. But from the code in Proposed Changes, once we
> > > register
> > > > the Catalog,
> > > > we initialize it and open it. right?
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > ----- 原始邮件 -----
> > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > 收件人: "dev" <de...@flink.apache.org>
> > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > configuration
> > > > and asynchronous registration
> > > >
> > > > Hi Feng,
> > > >
> > > > Thanks for your effort! +1 for the proposal.
> > > >
> > > > One of the major changes is that current design will provide
> > > > Map<String,Catalog> catalogs as a snapshot instead of a cache, which
> > > means
> > > > once it has been initialized, any changes done by other sessions will
> > not
> > > > affect it. Point 6 described follow-up options for further
> improvement.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> > wrote:
> > > >
> > > > > Hi all, I would like to update you on the latest progress of the
> > FLIP.
> > > > >
> > > > >
> > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang
> > > and I
> > > > > had an offline discussion regarding the overall solution for Flink
> > > > > CatalogStore. We have reached a consensus and I have updated the
> > final
> > > > > solution in FLIP.
> > > > >
> > > > > Next, let me briefly describe the entire design:
> > > > >
> > > > > 1.
> > > > >
> > > > > Introduce CatalogDescriptor to store catalog configuration
> similar
> > > to
> > > > > TableDescriptor.
> > > > > 2.
> > > > >
> > > > > The two key functions of CatalogStore - void storeCatalog(String
> > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > getCatalog(String)
> > > > > will both use CatalogDescriptor instead of Catalog instance.
> This
> > > way,
> > > > > CatalogStore will only be responsible for saving and retrieving
> > > > catalog
> > > > > configurations without having to initialize catalogs.
> > > > > 3.
> > > > >
> > > > > The default registerCatalog(String catalogName, Catalog catalog)
> > > > > function in CatalogManager will be marked as deprecated.
> > > > > 4.
> > > > >
> > > > > A new function registerCatalog(String catalogName,
> > CatalogDescriptor
> > > > > catalog) will be added to serve as the default registration
> > function
> > > > for
> > > > > catalogs in CatalogManager.
> > > > > 5.
> > > > >
> > > > > Map<String,Catalog> catalogs in CataloManager will remain
> > unchanged
> > > > and
> > > > > save initialized catalogs.This means that deletion operations
> from
> > > one
> > > > > session won't synchronize with other sessions.
> > > > > 6.
> > > > >
> > > > > To support multi-session synchronization scenarios for deletions
> > > later
> > > > > on we should make Map<String,Catalog>catalogs configurable.There
> > may
> > > > be
> > > > > three possible situations:
> > > > >
> > > > > a.Default caching of all initialized catalogs
> > > > >
> > > > > b.Introduction of LRU cache logic which can automatically clear
> > > > > long-unused catalogs.
> > > > >
> > > > > c.No caching of any instances; each call to getCatalog creates a
> > new
> > > > > instance.
> > > > >
> > > > >
> > > > > This is the document for discussion:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > >
> > > > > This is the final proposal document:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > >
> > > > >
> > > > > Thank you very much for your attention and suggestions on this
> > FLIP. A
> > > > > special thanks to Hang Ruan for his suggestions on the entire
> design
> > > and
> > > > > organizing offline discussions.
> > > > >
> > > > > If you have any further suggestions or feedback about this FLIP
> > please
> > > > feel
> > > > > free to share.
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Feng
> > > > >
> > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge <jing@ververica.com.invalid
> >
> > > > wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > Thanks for improving the FLIP. It looks good to me. We could
> still
> > > > > > reconsider in the future how to provide more common built-in
> cache
> > > > > > functionality in CatalogManager so that not every CatalogSotre
> > > > > > implementation has to take care of it.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi Jing,
> > > > > > >
> > > > > > > Thanks for your reply.
> > > > > > >
> > > > > > > > There might be more such issues. I would suggest you
> > completely
> > > > walk
> > > > > > > through the FLIP again and fix those issues
> > > > > > >
> > > > > > > I am very sorry for my carelessness and at the same time, I
> > greatly
> > > > > > > appreciate your careful review. I have thoroughly checked the
> > > entire
> > > > > FLIP
> > > > > > > and made corrections to these issues.
> > > > > > >
> > > > > > >
> > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > CatalogManager
> > > > > > > could work without Optional CatalogStore being configured.
> > > > > > >
> > > > > > > Yes, in the original design, CatalogStore was not necessary
> > because
> > > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> > catalog
> > > > > > > instances.
> > > > > > > However, this caused inconsistency issues. Therefore, I
> modified
> > > this
> > > > > > part
> > > > > > > of the design and removed Map<String, Catalog> catalogs from
> > > > > > > CatalogManager.
> > > > > > > At the same time, InMemoryCatalog will serve as the default
> > > > > CatalogStore
> > > > > > > to save catalogs in memory and replace the functionality of
> > > > > > > Map<String,Catalog>catalogs.
> > > > > > > The previous plan that kept Map<String,Catalog>catalogs has
> been
> > > > moved
> > > > > to
> > > > > > > Rejected Alternatives.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > There are still many places contain inconsistent content,
> e.g.
> > > > > > > >
> > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > 2. The java comment of the method registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog catalog) in CatalogManager does not tell what the
> > method
> > > is
> > > > > > > doing.
> > > > > > > >
> > > > > > > >
> > > > > > > > There might be more such issues. I would suggest you
> completely
> > > > walk
> > > > > > > > through the FLIP again and fix those issues.
> > > > > > > >
> > > > > > > > About the inMemoryCatalogStore, do you mean that you will
> build
> > > the
> > > > > > cache
> > > > > > > > functionality in the CatalogStore? This is a very different
> > > design
> > > > > > > concept
> > > > > > > > from what the current FLIP described. If I am not mistaken,
> > with
> > > > the
> > > > > > > > current FLIP design, CatalogManager could work without
> Optional
> > > > > > > > CatalogStore being configured. That is the reason why I
> > mentioned
> > > > in
> > > > > > the
> > > > > > > > last email that the example code wrt the Optional
> CatalogStore
> > is
> > > > not
> > > > > > > > correct.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > jinfeng1995@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jing
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > >
> > > > > > > > > Thank you very much for the careful review. I have already
> > made
> > > > the
> > > > > > > > > relevant changes.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > scenarios, I
> > > > > > > am
> > > > > > > > > not
> > > > > > > > > sure if this is the intended behaviour
> > > > > > > > >
> > > > > > > > > Currently, there are conflicts in multiple scenarios with
> the
> > > > > current
> > > > > > > > > design. I am thinking whether we should remove 'Map<String,
> > > > > Catalog>'
> > > > > > > and
> > > > > > > > > make Cache the default behavior of InMemoryCatalogStore.
> This
> > > > way,
> > > > > > > users
> > > > > > > > > can implement their own CatalogStore to achieve
> > multi-instance
> > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for working on the FLIP. There are still some NIT
> > > issues
> > > > > in
> > > > > > > the
> > > > > > > > > FLIP
> > > > > > > > > > like:
> > > > > > > > > >
> > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > > > > > CatalogStore
> > > > > > > > > > instead of Optional in the code example. It should be
> fine
> > to
> > > > use
> > > > > > it
> > > > > > > as
> > > > > > > > > > pseudo code for now and update it after you submit the
> PR.
> > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> rejected
> > > > > section
> > > > > > > > which
> > > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > > >
> > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > scenarios,
> > > > > I
> > > > > > am
> > > > > > > > not
> > > > > > > > > > sure if this is the intended behaviour. If Map<String,
> > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > is
> > > > > > > > > > used as a cache, it should be invalid, once the related
> > > catalog
> > > > > has
> > > > > > > > been
> > > > > > > > > > removed from the CatalogStore by another instance. Did I
> > miss
> > > > > > > > something?
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > jinfeng1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > >
> > > > > > > > > > > @Jing
> > > > > > > > > > >
> > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > I think this is a good function name, I have updated it
> > in
> > > > the
> > > > > > > > > > > documentation.
> > > > > > > > > > >
> > > > > > > > > > > > Some common cache features should be implemented
> > > > > > > > > > > Thank you for the suggestion. If alternative 1 turns
> out
> > to
> > > > be
> > > > > > more
> > > > > > > > > > > appropriate later, I will improve this part of the
> > content.
> > > > > > > > > > >
> > > > > > > > > > > > As the above discussion moves forward, the option 2
> > > > solution
> > > > > > > looks
> > > > > > > > > more
> > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > Yes, after discussing with Shammon offline, we think
> that
> > > > > > solution
> > > > > > > 2
> > > > > > > > > > might
> > > > > > > > > > > be more suitable and also avoid any inconsistency
> issues.
> > > > > > > > > > >
> > > > > > > > > > > > There are some inconsistent descriptions in the
> > content.
> > > > > Would
> > > > > > > you
> > > > > > > > > > like
> > > > > > > > > > > to clean them up?
> > > > > > > > > > > I will do my best to improve the document and
> appreciate
> > > your
> > > > > > > > > > suggestions.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > @Shammon
> > > > > > > > > > > > can you put the unselected option in `Rejected
> > > > Alternatives`
> > > > > > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > zjureel@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > >
> > > > > > > > > > > > I found there are two options in `Proposed Changes`,
> > can
> > > > you
> > > > > > put
> > > > > > > > the
> > > > > > > > > > > > unselected option in `Rejected Alternatives`? I think
> > > this
> > > > > may
> > > > > > > help
> > > > > > > > > us
> > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Shammon FY
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for raising this FLIP. I am still confused
> > after
> > > > > > > > completely
> > > > > > > > > > > > reading
> > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > addCatalog()
> > > > > have
> > > > > > > no
> > > > > > > > > big
> > > > > > > > > > > > > difference based on their names. One of them is
> > > > responsible
> > > > > > for
> > > > > > > > > data
> > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> > catalogs
> > > is
> > > > > > used
> > > > > > > > as a
> > > > > > > > > > > cache
> > > > > > > > > > > > > and catalogStore is used for data persistence. I
> > would
> > > > > > suggest
> > > > > > > > > > > describing
> > > > > > > > > > > > > their purpose conceptually and clearly in the FLIP.
> > > Some
> > > > > > common
> > > > > > > > > cache
> > > > > > > > > > > > > features should be implemented, i.e. data in the
> > cache
> > > > and
> > > > > > the
> > > > > > > > > store
> > > > > > > > > > > > should
> > > > > > > > > > > > > be consistent. Same Catalog instance should be
> found
> > in
> > > > the
> > > > > > > store
> > > > > > > > > and
> > > > > > > > > > > in
> > > > > > > > > > > > > the cache(either it has been initialized or it will
> > be
> > > > lazy
> > > > > > > > > > > initialized)
> > > > > > > > > > > > > for the same catalog name. The consistency will be
> > > taken
> > > > > care
> > > > > > > of
> > > > > > > > > > while
> > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > 3. As the above discussion moves forward, the
> option
> > 2
> > > > > > solution
> > > > > > > > > looks
> > > > > > > > > > > > more
> > > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> > issues
> > > > > > > mentioned
> > > > > > > > > > > > > previously with option 1 are not solved yet. Do you
> > > still
> > > > > > want
> > > > > > > to
> > > > > > > > > > > propose
> > > > > > > > > > > > > both options and ask for suggestions for both of
> > them?
> > > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > > inconsistent
> > > > > > > > > > descriptions
> > > > > > > > > > > > in
> > > > > > > > > > > > > the content. Would you like to clean them up?
> > Thanks!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > jinfeng1995@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for your response, and I completely
> agree
> > > > with
> > > > > > your
> > > > > > > > > point
> > > > > > > > > > > of
> > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > Initially, I may have over complicated the whole
> > > issue.
> > > > > > First
> > > > > > > > and
> > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > we need to consider the persistence of the
> > Catalog's
> > > > > > > > > Configuration.
> > > > > > > > > > > > > > If we only need to provide persistence for
> Catalog
> > > > > > > > Configuration,
> > > > > > > > > > we
> > > > > > > > > > > > can
> > > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > > interface.
> > > > > > > > > > > > > > This method can convert a Catalog instance to a
> > > > > Map<String,
> > > > > > > > > String>
> > > > > > > > > > > > > > properties, and the default implementation will
> > throw
> > > > an
> > > > > > > > > exception.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > * Returns a map containing the properties of
> the
> > > > > catalog
> > > > > > > > > object.
> > > > > > > > > > > > > > *
> > > > > > > > > > > > > > * @return a map containing the properties of
> the
> > > > > catalog
> > > > > > > > > object
> > > > > > > > > > > > > > * @throws UnsupportedOperationException if the
> > > > > > > implementing
> > > > > > > > > > class
> > > > > > > > > > > > does
> > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > * the default implementation of this method
> > > > > > > > > > > > > > */
> > > > > > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > > > > > throw new
> UnsupportedOperationException("Please
> > > > > > implement
> > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. If the user has configured a CatalogStore,
> the
> > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > will be called when registering a Catalog
> instance
> > > with
> > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > catalog).
> > > > The
> > > > > > > > Catalog
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > will be converted to a Map<String, String>
> > properties
> > > > and
> > > > > > > saved
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > CatalogStore. If some Catalog instances do not
> > > > implement
> > > > > > this
> > > > > > > > > > method,
> > > > > > > > > > > > an
> > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. If the user does not use a CatalogStore, the
> > > > > > > > > toConfiguration()
> > > > > > > > > > > > method
> > > > > > > > > > > > > > will not be called, ensuring consistency with the
> > > > > original
> > > > > > > > > process.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog> catalogs
> > and
> > > > the
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > at
> > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > we can start from the Catalog itself and provide
> a
> > > > > > dedicated
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > ConfigurationCatalog
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > public class ConfigurationCatalog implements
> > Catalog
> > > {
> > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > properties)
> > > {
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > > zjureel@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have no questions about the functionality of
> > > > > > > > `CatalogStore`,
> > > > > > > > > > but
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > different operations of multiple
> > `registerCatalog`
> > > or
> > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > `CatalogManager`. The implementation in Trino
> is
> > > also
> > > > > the
> > > > > > > > same,
> > > > > > > > > > the
> > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > `createCatalog`,
> > > > > > > which
> > > > > > > > > > will
> > > > > > > > > > > > save
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think we don't need to add another
> > > > `registerCatalog`
> > > > > or
> > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> catalogs`
> > > or
> > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > catalogStore`. As you mentioned before, the
> > > > > `Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How
> > about
> > > > > > saving
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> `CatalogStore
> > > > > > > > catalogStore`
> > > > > > > > > > > > > together
> > > > > > > > > > > > > > > when it is registered or added in
> > `CatalogManager`?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you for your reply. I am very sorry for
> > the
> > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> > this
> > > > FLIP
> > > > > > > > > > > > > > > > Yes, there was indeed such a discussion
> before.
> > > > > > However,
> > > > > > > > in
> > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > > CatalogManager
> > > > > > > > > itself
> > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > need to change much. *We cannot only persist
> > > > Catalog
> > > > > > > > > instances
> > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > not be enough to save Catalogs, because in
> the
> > > end
> > > > we
> > > > > > > still
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > the configurations corresponding to the
> Catalog
> > > > > > > instances.
> > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> interface
> > > for
> > > > > > > > > > configuration
> > > > > > > > > > > > > > > > persistence. Regarding this part, I also took
> > > > > > inspiration
> > > > > > > > > from
> > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with
> the
> > > > same
> > > > > > name
> > > > > > > > > > looks a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > Thank you for the suggestion. I will provide
> a
> > > > > detailed
> > > > > > > > > > > description
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > code examples for this part, and add it to
> the
> > > > FLIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still
> > > looks
> > > > > > > confused
> > > > > > > > > to
> > > > > > > > > > > me.
> > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > this method is for storing catalog to
> > > CatalogStore,
> > > > > how
> > > > > > > > about
> > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> opinion,
> > > you
> > > > > can
> > > > > > > > decide
> > > > > > > > > > to
> > > > > > > > > > > > take
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me. I
> > don't
> > > > see
> > > > > > any
> > > > > > > > > > problem
> > > > > > > > > > > > with
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which
> method
> > > will
> > > > > be
> > > > > > > > > called?
> > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > wondering
> > > > > > whether
> > > > > > > > user
> > > > > > > > > > can
> > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > For CREATE CATALOG, my original design was to
> > add
> > > > it
> > > > > > > > directly
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> > existing
> > > > > > logic.
> > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > we can do both: save the configuration to the
> > > > > > > CatalogStore
> > > > > > > > > and
> > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> > default
> > > > > > > > > implmentation
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > I think it is necessary, otherwise we would
> > need
> > > to
> > > > > > > > introduce
> > > > > > > > > > an
> > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > Map to store the configuration for lazy
> > loading.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > catalog.
> > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > > catalog)
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > made into an asynchronous interface because
> > > Catalog
> > > > > is
> > > > > > > > > already
> > > > > > > > > > an
> > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It looks more like lazy initialization for
> > > > catalog
> > > > > > than
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> > should
> > > be
> > > > > > lazy
> > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > instead of asynchronous registration. I have
> > > > already
> > > > > > > > updated
> > > > > > > > > > the
> > > > > > > > > > > > > title
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > > > > > zjureel@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> > method
> > > in
> > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > confuse users whether a method named
> > > `addCatalog`
> > > > > or
> > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > And as you mentioned, the memory catalog
> is a
> > > > > > `cache`,
> > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> > this
> > > > > FLIP.
> > > > > > > > Please
> > > > > > > > > > > > correct
> > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > > discussion
> > > > > is
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> > implement
> > > it
> > > > > for
> > > > > > > > > > different
> > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think there is a gap between the current
> > FLIP
> > > > > > design
> > > > > > > > and
> > > > > > > > > > that
> > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > What about the proposal of the discussion
> in
> > > > thread
> > > > > > > [1] ?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > >
> > > > > > >
> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with
> > the
> > > > > same
> > > > > > > name
> > > > > > > > > > > looks a
> > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > complex to me. I think it'll be better to
> > > > explain
> > > > > > it
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > java
> > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > these methods and give a brief example in
> > > this
> > > > > > FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> still
> > > > looks
> > > > > > > > confused
> > > > > > > > > > to
> > > > > > > > > > > > me.
> > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > CatalogStore,
> > > > > > > how
> > > > > > > > > > about
> > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > opinion,
> > > > > you
> > > > > > > can
> > > > > > > > > > decide
> > > > > > > > > > > > to
> > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which
> > method
> > > > > will
> > > > > > be
> > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > > wondering
> > > > > > > > whether
> > > > > > > > > > user
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3: Is it really neccessary to provide a
> > > default
> > > > > > > > > > implmentation
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > catalog.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > > CATALOG,
> > > > > the
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > > > > > If asynchronous registration is the
> default
> > > > > > behavior,
> > > > > > > > it
> > > > > > > > > > > there
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > > registration
> > > > > > just
> > > > > > > > > like
> > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > `registerCatalog`
> > > > > > > be
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > IIUC, in asynchronous registration, it
> may
> > > well
> > > > > > that
> > > > > > > > > CREATE
> > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > executes successfully, but then the
> > following
> > > > > > CREATE
> > > > > > > > > TABLE
> > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > > I think it's a break change which should
> be
> > > > > > > highlighted
> > > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> registration, I
> > > > would
> > > > > > > like
> > > > > > > > to
> > > > > > > > > > > > expect
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > will be an executor to open or register
> > > catalog
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > "the Catalog instance will be initialized
> > if
> > > it
> > > > > has
> > > > > > > not
> > > > > > > > > > been
> > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > yet. If the initialization process fails,
> > > these
> > > > > > > > > statements
> > > > > > > > > > > will
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> for
> > > > > catalog
> > > > > > > than
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > > persistence
> > > > > of
> > > > > > > > > Catalog
> > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> > interest
> > > in
> > > > > > this
> > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > How to handle a catalog with the same
> > name
> > > > that
> > > > > > > > exists
> > > > > > > > > > for
> > > > > > > > > > > > both
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I believe this is a crucial point. Based
> on
> > > my
> > > > > > > current
> > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > understanding, the Map<String, Catalog>
> > > > catalogs
> > > > > > will
> > > > > > > > > serve
> > > > > > > > > > > as
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > for instantiated catalogs and have the
> > > highest
> > > > > > > > priority.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There are three methods that can affect
> the
> > > > > > > Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. registerCatalog(String catalogName,
> > > Catalog
> > > > > > > catalog)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This method puts the catalog instance
> into
> > > the
> > > > > > > > > Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This method removes the catalog instance
> > > > > > > corresponding
> > > > > > > > to
> > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This method first retrieves the
> > corresponding
> > > > > > catalog
> > > > > > > > > > > instance
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If the
> > > > catalog
> > > > > > > does
> > > > > > > > > not
> > > > > > > > > > > > exist,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > retrieves the corresponding configuration
> > > from
> > > > > the
> > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > initializes it, and puts the initialized
> > > > Catalog
> > > > > > > > instance
> > > > > > > > > > > into
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The following two methods only modify the
> > > > > > > configuration
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > Map<String,
> > > > > > String>
> > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This method saves the properties to the
> > > > > > catalogStore
> > > > > > > > and
> > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > whether there is a catalogName with the
> > same
> > > > > name.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > This method removes the specified
> > > configuration
> > > > > of
> > > > > > > the
> > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The following are possible conflict
> > > scenarios:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> > already
> > > > > > exists
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore but not in the Map<String,
> > > > Catalog>,
> > > > > > the
> > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> Catalog
> > > > > > catalog)
> > > > > > > > > method
> > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> > already
> > > > > > exists
> > > > > > > in
> > > > > > > > > > both
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> Catalog>,
> > > the
> > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method will
> > > fail.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> > already
> > > > > > exists
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> addCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > String> properties) method can directly
> > save
> > > > the
> > > > > > > > > properties
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > > > > > catalogName)
> > > > > > > > > method
> > > > > > > > > > > > will
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > use the new properties for initialization
> > > > because
> > > > > > the
> > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > catalog instance already exists in
> catalogs
> > > and
> > > > > > will
> > > > > > > be
> > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > Therefore, using the
> > unregisterCatalog(String
> > > > > > > > > catalogName)
> > > > > > > > > > > > method
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > remove the instance corresponding to the
> > > > original
> > > > > > > > > > catalogName
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > > `Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > > Map<String,
> > > > > > > > String>
> > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This could potentially lead to
> confusion. I
> > > > > suggest
> > > > > > > > > > changing
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > method name, perhaps to addCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > String> properties), as previously
> > mentioned
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > catalogName,Catalog
> > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Since a catalog is already an instance,
> > > adding
> > > > > the
> > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > parameter to the registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > method may not necessarily result in lazy
> > > > > > > > initialization.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think encryption is necessary, but
> > perhaps
> > > > the
> > > > > > > > > encryption
> > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > should be implemented in the CatalogStore
> > > when
> > > > > the
> > > > > > > data
> > > > > > > > > is
> > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > saved. For instance, the FileCatalogStore
> > > could
> > > > > > > encrypt
> > > > > > > > > the
> > > > > > > > > > > > data
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > MemoryCatalogStore
> > > > > > > > does
> > > > > > > > > > not
> > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Do we really need the
> > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > MemoryCatalogStore
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > > configurations
> > > > > > in
> > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > > asynchronous
> > > > > > > loading
> > > > > > > > > of
> > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > What if asynchronous registration
> failed?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This is also a critical concern. When
> > > > executing
> > > > > > DDL,
> > > > > > > > > DQL,
> > > > > > > > > > or
> > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > statements that reference a specific
> > Catalog,
> > > > the
> > > > > > > > Catalog
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > will be initialized if it has not been
> > > > > initialized
> > > > > > > yet.
> > > > > > > > > If
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > initialization process fails, these
> > > statements
> > > > > will
> > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs
> and
> > > > > > > > CatalogStore
> > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the CatalogManager all have the same
> > catalog
> > > > > name,
> > > > > > > but
> > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This issue can be resolved by referring
> to
> > > the
> > > > > > > previous
> > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > key principle is that the Catalog that
> has
> > > been
> > > > > > most
> > > > > > > > > > recently
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > will be given priority for subsequent
> use.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun
> Ye
> > <
> > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> > believe
> > > > this
> > > > > > > Flip
> > > > > > > > > > could
> > > > > > > > > > > be
> > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > for users. The firm I work for also
> > manages a
> > > > lot
> > > > > > of
> > > > > > > > > > > catalogs,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > submission of tasks becomes slow because
> of
> > > > > > loading a
> > > > > > > > > > number
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > We obtain the catalogs in the user's
> Flink
> > > SQL
> > > > > > > through
> > > > > > > > > > > regular
> > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > to avoid loading all catalogs to improve
> > the
> > > > > speed
> > > > > > of
> > > > > > > > > task
> > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > > questions:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. When creating a catalog with CREATE
> > > > CATALOG,
> > > > > > the
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration method is used by default.
> > What
> > > if
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > failed? And how to implement asynchronous
> > > > > > > registration?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. I also have the same question with
> > > > Shammon’s
> > > > > > > first
> > > > > > > > > > > > comment.
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > CatalogStore
> > > > > > > > > catalogstore
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogManager all have the same catalog
> > > name,
> > > > > but
> > > > > > > > > > > correponding
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > different catalog instances. For example:
> > > > > > catalog1->
> > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in
> > the
> > > > > > > > > catalogstore.
> > > > > > > > > > > > Would
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > happen? Maybe we should define `catalogs`
> > and
> > > > > > > > > > `catalogstore`
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > ruanhang1993@gmail.com
> > > > > > > > >
> > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Shammon FY <zj...@gmail.com>.
Hi feng,
Thanks for updating, I have some minor comments
1. The modification of `CatalogManager` should not be in `Public
Interfaces`, it is not a public interface.
2. `@PublicEvolving` should be added for `CatalogStore` and
`CatalogStoreFactory`
3. The code `Optional<Catalog> optionalDescriptor =
catalogStore.get(catalogName);` in the `CatalogManager` should be
`Optional<CatalogDescriptor> optionalDescriptor =
catalogStore.get(catalogName);`
Best,
Shammon FY
On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
> Hi, Feng
>
> Thanks for driving this FLIP, this proposal is very useful for catalog
> management.
> I have some small questions:
>
> 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we need
> to provide a default implementation?
> 2. If we get Catalog from CatalogStore, after initializing it, whether we
> put it to Map<String, Catalog> catalogs again?
> 3. Regarding the options `sql.catalog.store.type` and
> `sql.catalog.store.file.path`, how about renaming them to
> `catalog.store.type` and `catalog.store.path`?
>
> Best,
> Ron
>
> Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
>
> > Hi yuxia
> >
> > > But from the code in Proposed Changes, once we register the Catalog,
> we
> > initialize it and open it. right?
> >
> > Yes, In order to avoid inconsistent semantics of the original CREATE
> > CATALOG DDL, Catalog will be directly initialized in registerCatalog so
> > that parameter validation can be performed.
> >
> > In the current design, lazy initialization is mainly reflected in
> > getCatalog. If CatalogStore has already saved some catalog
> configurations,
> > only initialization is required in getCatalog.
> >
> >
> > Best,
> > Feng
> >
> > On Mon, May 29, 2023 at 8:27 PM yuxia <lu...@alumni.sjtu.edu.cn>
> wrote:
> >
> > > Hi, Feng.
> > > I'm trying to understanding the meaning of *lazy initialization*. If
> i'm
> > > wrong, please correct me.
> > >
> > > IIUC, lazy initialization means only you need to access the catalog,
> then
> > > you initialize it. But from the code in Proposed Changes, once we
> > register
> > > the Catalog,
> > > we initialize it and open it. right?
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > ----- 原始邮件 -----
> > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > 收件人: "dev" <de...@flink.apache.org>
> > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> configuration
> > > and asynchronous registration
> > >
> > > Hi Feng,
> > >
> > > Thanks for your effort! +1 for the proposal.
> > >
> > > One of the major changes is that current design will provide
> > > Map<String,Catalog> catalogs as a snapshot instead of a cache, which
> > means
> > > once it has been initialized, any changes done by other sessions will
> not
> > > affect it. Point 6 described follow-up options for further improvement.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> wrote:
> > >
> > > > Hi all, I would like to update you on the latest progress of the
> FLIP.
> > > >
> > > >
> > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang
> > and I
> > > > had an offline discussion regarding the overall solution for Flink
> > > > CatalogStore. We have reached a consensus and I have updated the
> final
> > > > solution in FLIP.
> > > >
> > > > Next, let me briefly describe the entire design:
> > > >
> > > > 1.
> > > >
> > > > Introduce CatalogDescriptor to store catalog configuration similar
> > to
> > > > TableDescriptor.
> > > > 2.
> > > >
> > > > The two key functions of CatalogStore - void storeCatalog(String
> > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > getCatalog(String)
> > > > will both use CatalogDescriptor instead of Catalog instance. This
> > way,
> > > > CatalogStore will only be responsible for saving and retrieving
> > > catalog
> > > > configurations without having to initialize catalogs.
> > > > 3.
> > > >
> > > > The default registerCatalog(String catalogName, Catalog catalog)
> > > > function in CatalogManager will be marked as deprecated.
> > > > 4.
> > > >
> > > > A new function registerCatalog(String catalogName,
> CatalogDescriptor
> > > > catalog) will be added to serve as the default registration
> function
> > > for
> > > > catalogs in CatalogManager.
> > > > 5.
> > > >
> > > > Map<String,Catalog> catalogs in CataloManager will remain
> unchanged
> > > and
> > > > save initialized catalogs.This means that deletion operations from
> > one
> > > > session won't synchronize with other sessions.
> > > > 6.
> > > >
> > > > To support multi-session synchronization scenarios for deletions
> > later
> > > > on we should make Map<String,Catalog>catalogs configurable.There
> may
> > > be
> > > > three possible situations:
> > > >
> > > > a.Default caching of all initialized catalogs
> > > >
> > > > b.Introduction of LRU cache logic which can automatically clear
> > > > long-unused catalogs.
> > > >
> > > > c.No caching of any instances; each call to getCatalog creates a
> new
> > > > instance.
> > > >
> > > >
> > > > This is the document for discussion:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > >
> > > > This is the final proposal document:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >
> > > >
> > > > Thank you very much for your attention and suggestions on this
> FLIP. A
> > > > special thanks to Hang Ruan for his suggestions on the entire design
> > and
> > > > organizing offline discussions.
> > > >
> > > > If you have any further suggestions or feedback about this FLIP
> please
> > > feel
> > > > free to share.
> > > >
> > > >
> > > > Best,
> > > >
> > > > Feng
> > > >
> > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > > >
> > > > > Hi Feng,
> > > > >
> > > > > Thanks for improving the FLIP. It looks good to me. We could still
> > > > > reconsider in the future how to provide more common built-in cache
> > > > > functionality in CatalogManager so that not every CatalogSotre
> > > > > implementation has to take care of it.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Jing,
> > > > > >
> > > > > > Thanks for your reply.
> > > > > >
> > > > > > > There might be more such issues. I would suggest you
> completely
> > > walk
> > > > > > through the FLIP again and fix those issues
> > > > > >
> > > > > > I am very sorry for my carelessness and at the same time, I
> greatly
> > > > > > appreciate your careful review. I have thoroughly checked the
> > entire
> > > > FLIP
> > > > > > and made corrections to these issues.
> > > > > >
> > > > > >
> > > > > > > If I am not mistaken, with the current FLIP design,
> > > CatalogManager
> > > > > > could work without Optional CatalogStore being configured.
> > > > > >
> > > > > > Yes, in the original design, CatalogStore was not necessary
> because
> > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> catalog
> > > > > > instances.
> > > > > > However, this caused inconsistency issues. Therefore, I modified
> > this
> > > > > part
> > > > > > of the design and removed Map<String, Catalog> catalogs from
> > > > > > CatalogManager.
> > > > > > At the same time, InMemoryCatalog will serve as the default
> > > > CatalogStore
> > > > > > to save catalogs in memory and replace the functionality of
> > > > > > Map<String,Catalog>catalogs.
> > > > > > The previous plan that kept Map<String,Catalog>catalogs has been
> > > moved
> > > > to
> > > > > > Rejected Alternatives.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > There are still many places contain inconsistent content, e.g.
> > > > > > >
> > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > 2. The java comment of the method registerCatalog(String
> > > catalogName,
> > > > > > > Catalog catalog) in CatalogManager does not tell what the
> method
> > is
> > > > > > doing.
> > > > > > >
> > > > > > >
> > > > > > > There might be more such issues. I would suggest you completely
> > > walk
> > > > > > > through the FLIP again and fix those issues.
> > > > > > >
> > > > > > > About the inMemoryCatalogStore, do you mean that you will build
> > the
> > > > > cache
> > > > > > > functionality in the CatalogStore? This is a very different
> > design
> > > > > > concept
> > > > > > > from what the current FLIP described. If I am not mistaken,
> with
> > > the
> > > > > > > current FLIP design, CatalogManager could work without Optional
> > > > > > > CatalogStore being configured. That is the reason why I
> mentioned
> > > in
> > > > > the
> > > > > > > last email that the example code wrt the Optional CatalogStore
> is
> > > not
> > > > > > > correct.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> jinfeng1995@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jing
> > > > > > > >
> > > > > > > >
> > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > >
> > > > > > > > Thank you very much for the careful review. I have already
> made
> > > the
> > > > > > > > relevant changes.
> > > > > > > >
> > > > > > > >
> > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > scenarios, I
> > > > > > am
> > > > > > > > not
> > > > > > > > sure if this is the intended behaviour
> > > > > > > >
> > > > > > > > Currently, there are conflicts in multiple scenarios with the
> > > > current
> > > > > > > > design. I am thinking whether we should remove 'Map<String,
> > > > Catalog>'
> > > > > > and
> > > > > > > > make Cache the default behavior of InMemoryCatalogStore. This
> > > way,
> > > > > > users
> > > > > > > > can implement their own CatalogStore to achieve
> multi-instance
> > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng
> > > > > > > >
> > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > Thanks for working on the FLIP. There are still some NIT
> > issues
> > > > in
> > > > > > the
> > > > > > > > FLIP
> > > > > > > > > like:
> > > > > > > > >
> > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > > > > CatalogStore
> > > > > > > > > instead of Optional in the code example. It should be fine
> to
> > > use
> > > > > it
> > > > > > as
> > > > > > > > > pseudo code for now and update it after you submit the PR.
> > > > > > > > > 2. addCatalog(...) is still used somewhere in the rejected
> > > > section
> > > > > > > which
> > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > >
> > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > scenarios,
> > > > I
> > > > > am
> > > > > > > not
> > > > > > > > > sure if this is the intended behaviour. If Map<String,
> > Catalog>
> > > > > > > catalogs
> > > > > > > > is
> > > > > > > > > used as a cache, it should be invalid, once the related
> > catalog
> > > > has
> > > > > > > been
> > > > > > > > > removed from the CatalogStore by another instance. Did I
> miss
> > > > > > > something?
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > jinfeng1995@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > Thanks for your reply.
> > > > > > > > > >
> > > > > > > > > > @Jing
> > > > > > > > > >
> > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > I think this is a good function name, I have updated it
> in
> > > the
> > > > > > > > > > documentation.
> > > > > > > > > >
> > > > > > > > > > > Some common cache features should be implemented
> > > > > > > > > > Thank you for the suggestion. If alternative 1 turns out
> to
> > > be
> > > > > more
> > > > > > > > > > appropriate later, I will improve this part of the
> content.
> > > > > > > > > >
> > > > > > > > > > > As the above discussion moves forward, the option 2
> > > solution
> > > > > > looks
> > > > > > > > more
> > > > > > > > > > like a replacement of option 1
> > > > > > > > > > Yes, after discussing with Shammon offline, we think that
> > > > > solution
> > > > > > 2
> > > > > > > > > might
> > > > > > > > > > be more suitable and also avoid any inconsistency issues.
> > > > > > > > > >
> > > > > > > > > > > There are some inconsistent descriptions in the
> content.
> > > > Would
> > > > > > you
> > > > > > > > > like
> > > > > > > > > > to clean them up?
> > > > > > > > > > I will do my best to improve the document and appreciate
> > your
> > > > > > > > > suggestions.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > @Shammon
> > > > > > > > > > > can you put the unselected option in `Rejected
> > > Alternatives`
> > > > > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > zjureel@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your update.
> > > > > > > > > > >
> > > > > > > > > > > I found there are two options in `Proposed Changes`,
> can
> > > you
> > > > > put
> > > > > > > the
> > > > > > > > > > > unselected option in `Rejected Alternatives`? I think
> > this
> > > > may
> > > > > > help
> > > > > > > > us
> > > > > > > > > > > better understand your proposal
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Shammon FY
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for raising this FLIP. I am still confused
> after
> > > > > > > completely
> > > > > > > > > > > reading
> > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > addCatalog()
> > > > have
> > > > > > no
> > > > > > > > big
> > > > > > > > > > > > difference based on their names. One of them is
> > > responsible
> > > > > for
> > > > > > > > data
> > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> catalogs
> > is
> > > > > used
> > > > > > > as a
> > > > > > > > > > cache
> > > > > > > > > > > > and catalogStore is used for data persistence. I
> would
> > > > > suggest
> > > > > > > > > > describing
> > > > > > > > > > > > their purpose conceptually and clearly in the FLIP.
> > Some
> > > > > common
> > > > > > > > cache
> > > > > > > > > > > > features should be implemented, i.e. data in the
> cache
> > > and
> > > > > the
> > > > > > > > store
> > > > > > > > > > > should
> > > > > > > > > > > > be consistent. Same Catalog instance should be found
> in
> > > the
> > > > > > store
> > > > > > > > and
> > > > > > > > > > in
> > > > > > > > > > > > the cache(either it has been initialized or it will
> be
> > > lazy
> > > > > > > > > > initialized)
> > > > > > > > > > > > for the same catalog name. The consistency will be
> > taken
> > > > care
> > > > > > of
> > > > > > > > > while
> > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > 3. As the above discussion moves forward, the option
> 2
> > > > > solution
> > > > > > > > looks
> > > > > > > > > > > more
> > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> issues
> > > > > > mentioned
> > > > > > > > > > > > previously with option 1 are not solved yet. Do you
> > still
> > > > > want
> > > > > > to
> > > > > > > > > > propose
> > > > > > > > > > > > both options and ask for suggestions for both of
> them?
> > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > inconsistent
> > > > > > > > > descriptions
> > > > > > > > > > > in
> > > > > > > > > > > > the content. Would you like to clean them up?
> Thanks!
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > jinfeng1995@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you for your response, and I completely agree
> > > with
> > > > > your
> > > > > > > > point
> > > > > > > > > > of
> > > > > > > > > > > > > view.
> > > > > > > > > > > > > Initially, I may have over complicated the whole
> > issue.
> > > > > First
> > > > > > > and
> > > > > > > > > > > > foremost,
> > > > > > > > > > > > > we need to consider the persistence of the
> Catalog's
> > > > > > > > Configuration.
> > > > > > > > > > > > > If we only need to provide persistence for Catalog
> > > > > > > Configuration,
> > > > > > > > > we
> > > > > > > > > > > can
> > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > interface.
> > > > > > > > > > > > > This method can convert a Catalog instance to a
> > > > Map<String,
> > > > > > > > String>
> > > > > > > > > > > > > properties, and the default implementation will
> throw
> > > an
> > > > > > > > exception.
> > > > > > > > > > > > >
> > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > /**
> > > > > > > > > > > > > * Returns a map containing the properties of the
> > > > catalog
> > > > > > > > object.
> > > > > > > > > > > > > *
> > > > > > > > > > > > > * @return a map containing the properties of the
> > > > catalog
> > > > > > > > object
> > > > > > > > > > > > > * @throws UnsupportedOperationException if the
> > > > > > implementing
> > > > > > > > > class
> > > > > > > > > > > does
> > > > > > > > > > > > > not override
> > > > > > > > > > > > > * the default implementation of this method
> > > > > > > > > > > > > */
> > > > > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > > > > throw new UnsupportedOperationException("Please
> > > > > implement
> > > > > > > > > > > > toProperties
> > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > }
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. If the user has configured a CatalogStore, the
> > > > > > > > > toConfiguration()
> > > > > > > > > > > > method
> > > > > > > > > > > > > will be called when registering a Catalog instance
> > with
> > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> catalog).
> > > The
> > > > > > > Catalog
> > > > > > > > > > > > instance
> > > > > > > > > > > > > will be converted to a Map<String, String>
> properties
> > > and
> > > > > > saved
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > CatalogStore. If some Catalog instances do not
> > > implement
> > > > > this
> > > > > > > > > method,
> > > > > > > > > > > an
> > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. If the user does not use a CatalogStore, the
> > > > > > > > toConfiguration()
> > > > > > > > > > > method
> > > > > > > > > > > > > will not be called, ensuring consistency with the
> > > > original
> > > > > > > > process.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. Saving both the Map<String, Catalog> catalogs
> and
> > > the
> > > > > > > > > CatalogStore
> > > > > > > > > > > at
> > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > we can start from the Catalog itself and provide a
> > > > > dedicated
> > > > > > > > > Catalog
> > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > ConfigurationCatalog
> > > > > > > > > > > > >
> > > > > > > > > > > > > public class ConfigurationCatalog implements
> Catalog
> > {
> > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> properties)
> > {
> > > > > > > > > > > > > }
> > > > > > > > > > > > > }
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > zjureel@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have no questions about the functionality of
> > > > > > > `CatalogStore`,
> > > > > > > > > but
> > > > > > > > > > > the
> > > > > > > > > > > > > > different operations of multiple
> `registerCatalog`
> > or
> > > > > > > > > > `storeCatalog`
> > > > > > > > > > > in
> > > > > > > > > > > > > > `CatalogManager`. The implementation in Trino is
> > also
> > > > the
> > > > > > > same,
> > > > > > > > > the
> > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > `createCatalog`,
> > > > > > which
> > > > > > > > > will
> > > > > > > > > > > save
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think we don't need to add another
> > > `registerCatalog`
> > > > or
> > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > to
> > > > > > > > > > > > > > save a catalog in `Map<String, Catalog> catalogs`
> > or
> > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > catalogStore`. As you mentioned before, the
> > > > `Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How
> about
> > > > > saving
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > in
> > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore
> > > > > > > catalogStore`
> > > > > > > > > > > > together
> > > > > > > > > > > > > > when it is registered or added in
> `CatalogManager`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > jinfeng1995@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you for your reply. I am very sorry for
> the
> > > > > > > > > > misunderstanding
> > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> this
> > > FLIP
> > > > > > > > > > > > > > > Yes, there was indeed such a discussion before.
> > > > > However,
> > > > > > > in
> > > > > > > > > > > > designing
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > CatalogManager
> > > > > > > > itself
> > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > need to change much. *We cannot only persist
> > > Catalog
> > > > > > > > instances
> > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > catalogName,
> > > > > > > Catalog
> > > > > > > > > > > catalog)
> > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > not be enough to save Catalogs, because in the
> > end
> > > we
> > > > > > still
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > save
> > > > > > > > > > > > > > > the configurations corresponding to the Catalog
> > > > > > instances.
> > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > decided to introduce the CatalogStore interface
> > for
> > > > > > > > > configuration
> > > > > > > > > > > > > > > persistence. Regarding this part, I also took
> > > > > inspiration
> > > > > > > > from
> > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with the
> > > same
> > > > > name
> > > > > > > > > looks a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > Thank you for the suggestion. I will provide a
> > > > detailed
> > > > > > > > > > description
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > code examples for this part, and add it to the
> > > FLIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still
> > looks
> > > > > > confused
> > > > > > > > to
> > > > > > > > > > me.
> > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > this method is for storing catalog to
> > CatalogStore,
> > > > how
> > > > > > > about
> > > > > > > > > > > > renaming
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > to `storeCatalog`? It's very personal opinion,
> > you
> > > > can
> > > > > > > decide
> > > > > > > > > to
> > > > > > > > > > > take
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me. I
> don't
> > > see
> > > > > any
> > > > > > > > > problem
> > > > > > > > > > > with
> > > > > > > > > > > > > it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which method
> > will
> > > > be
> > > > > > > > called?
> > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> wondering
> > > > > whether
> > > > > > > user
> > > > > > > > > can
> > > > > > > > > > > > add a
> > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > For CREATE CATALOG, my original design was to
> add
> > > it
> > > > > > > directly
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> existing
> > > > > logic.
> > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > we can do both: save the configuration to the
> > > > > > CatalogStore
> > > > > > > > and
> > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> default
> > > > > > > > implmentation
> > > > > > > > > > for
> > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > I think it is necessary, otherwise we would
> need
> > to
> > > > > > > introduce
> > > > > > > > > an
> > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > Map to store the configuration for lazy
> loading.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 4: About asynchronous registration for
> catalog.
> > > > > > > > > > > > > > > I don't think registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > > catalog)
> > > > > > > > > > > > can
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > made into an asynchronous interface because
> > Catalog
> > > > is
> > > > > > > > already
> > > > > > > > > an
> > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It looks more like lazy initialization for
> > > catalog
> > > > > than
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> should
> > be
> > > > > lazy
> > > > > > > > > > > registration
> > > > > > > > > > > > > > > instead of asynchronous registration. I have
> > > already
> > > > > > > updated
> > > > > > > > > the
> > > > > > > > > > > > title
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > > > > zjureel@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> method
> > in
> > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > confuse users whether a method named
> > `addCatalog`
> > > > or
> > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > And as you mentioned, the memory catalog is a
> > > > > `cache`,
> > > > > > I
> > > > > > > > > think
> > > > > > > > > > > the
> > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> this
> > > > FLIP.
> > > > > > > Please
> > > > > > > > > > > correct
> > > > > > > > > > > > > me
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > discussion
> > > > is
> > > > > > to
> > > > > > > > use
> > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> implement
> > it
> > > > for
> > > > > > > > > different
> > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think there is a gap between the current
> FLIP
> > > > > design
> > > > > > > and
> > > > > > > > > that
> > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > What about the proposal of the discussion in
> > > thread
> > > > > > [1] ?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with
> the
> > > > same
> > > > > > name
> > > > > > > > > > looks a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > complex to me. I think it'll be better to
> > > explain
> > > > > it
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > java
> > > > > > > > > > > > > doc
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > these methods and give a brief example in
> > this
> > > > > FLIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still
> > > looks
> > > > > > > confused
> > > > > > > > > to
> > > > > > > > > > > me.
> > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > CatalogStore,
> > > > > > how
> > > > > > > > > about
> > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > opinion,
> > > > you
> > > > > > can
> > > > > > > > > decide
> > > > > > > > > > > to
> > > > > > > > > > > > > take
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which
> method
> > > > will
> > > > > be
> > > > > > > > > called?
> > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > wondering
> > > > > > > whether
> > > > > > > > > user
> > > > > > > > > > > can
> > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3: Is it really neccessary to provide a
> > default
> > > > > > > > > implmentation
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > catalog.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > CATALOG,
> > > > the
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > > > > If asynchronous registration is the default
> > > > > behavior,
> > > > > > > it
> > > > > > > > > > there
> > > > > > > > > > > > any
> > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > registration
> > > > > just
> > > > > > > > like
> > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > `registerCatalog`
> > > > > > be
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > IIUC, in asynchronous registration, it may
> > well
> > > > > that
> > > > > > > > CREATE
> > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > executes successfully, but then the
> following
> > > > > CREATE
> > > > > > > > TABLE
> > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > I think it's a break change which should be
> > > > > > highlighted
> > > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > BTW, by saying asynchronous registration, I
> > > would
> > > > > > like
> > > > > > > to
> > > > > > > > > > > expect
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > will be an executor to open or register
> > catalog
> > > > in
> > > > > > the
> > > > > > > > > > > > background,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > "the Catalog instance will be initialized
> if
> > it
> > > > has
> > > > > > not
> > > > > > > > > been
> > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > yet. If the initialization process fails,
> > these
> > > > > > > > statements
> > > > > > > > > > will
> > > > > > > > > > > > not
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > It looks more like lazy initialization for
> > > > catalog
> > > > > > than
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > persistence
> > > > of
> > > > > > > > Catalog
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> interest
> > in
> > > > > this
> > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > How to handle a catalog with the same
> name
> > > that
> > > > > > > exists
> > > > > > > > > for
> > > > > > > > > > > both
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I believe this is a crucial point. Based on
> > my
> > > > > > current
> > > > > > > > > > personal
> > > > > > > > > > > > > > > > > understanding, the Map<String, Catalog>
> > > catalogs
> > > > > will
> > > > > > > > serve
> > > > > > > > > > as
> > > > > > > > > > > a
> > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > for instantiated catalogs and have the
> > highest
> > > > > > > priority.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are three methods that can affect the
> > > > > > Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. registerCatalog(String catalogName,
> > Catalog
> > > > > > catalog)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This method puts the catalog instance into
> > the
> > > > > > > > Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This method removes the catalog instance
> > > > > > corresponding
> > > > > > > to
> > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This method first retrieves the
> corresponding
> > > > > catalog
> > > > > > > > > > instance
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If the
> > > catalog
> > > > > > does
> > > > > > > > not
> > > > > > > > > > > exist,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > retrieves the corresponding configuration
> > from
> > > > the
> > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > initializes it, and puts the initialized
> > > Catalog
> > > > > > > instance
> > > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The following two methods only modify the
> > > > > > configuration
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> Map<String,
> > > > > String>
> > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This method saves the properties to the
> > > > > catalogStore
> > > > > > > and
> > > > > > > > > > checks
> > > > > > > > > > > > > > > > > whether there is a catalogName with the
> same
> > > > name.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > This method removes the specified
> > configuration
> > > > of
> > > > > > the
> > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The following are possible conflict
> > scenarios:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> already
> > > > > exists
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore but not in the Map<String,
> > > Catalog>,
> > > > > the
> > > > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > > > catalog)
> > > > > > > > method
> > > > > > > > > > can
> > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> already
> > > > > exists
> > > > > > in
> > > > > > > > > both
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore and the Map<String, Catalog>,
> > the
> > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method will
> > fail.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> already
> > > > > exists
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > > > > Map<String, Catalog>, the addCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > String> properties) method can directly
> save
> > > the
> > > > > > > > properties
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > > > > catalogName)
> > > > > > > > method
> > > > > > > > > > > will
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > use the new properties for initialization
> > > because
> > > > > the
> > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > catalog instance already exists in catalogs
> > and
> > > > > will
> > > > > > be
> > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > Therefore, using the
> unregisterCatalog(String
> > > > > > > > catalogName)
> > > > > > > > > > > method
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > remove the instance corresponding to the
> > > original
> > > > > > > > > catalogName
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > `Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > Map<String,
> > > > > > > String>
> > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This could potentially lead to confusion. I
> > > > suggest
> > > > > > > > > changing
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > method name, perhaps to addCatalog(String
> > > > > > catalogName,
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > String> properties), as previously
> mentioned
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > catalogName,Catalog
> > > > > > > > catalog,
> > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Since a catalog is already an instance,
> > adding
> > > > the
> > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > parameter to the registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > method may not necessarily result in lazy
> > > > > > > initialization.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think encryption is necessary, but
> perhaps
> > > the
> > > > > > > > encryption
> > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > should be implemented in the CatalogStore
> > when
> > > > the
> > > > > > data
> > > > > > > > is
> > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > saved. For instance, the FileCatalogStore
> > could
> > > > > > encrypt
> > > > > > > > the
> > > > > > > > > > > data
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > MemoryCatalogStore
> > > > > > > does
> > > > > > > > > not
> > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Do we really need the
> > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > MemoryCatalogStore
> > > > > > as
> > > > > > > > the
> > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > configurations
> > > > > in
> > > > > > > > > memory.
> > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > asynchronous
> > > > > > loading
> > > > > > > > of
> > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This is also a critical concern. When
> > > executing
> > > > > DDL,
> > > > > > > > DQL,
> > > > > > > > > or
> > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > statements that reference a specific
> Catalog,
> > > the
> > > > > > > Catalog
> > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > will be initialized if it has not been
> > > > initialized
> > > > > > yet.
> > > > > > > > If
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > initialization process fails, these
> > statements
> > > > will
> > > > > > not
> > > > > > > > be
> > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs and
> > > > > > > CatalogStore
> > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the CatalogManager all have the same
> catalog
> > > > name,
> > > > > > but
> > > > > > > > > > > > correponding
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > This issue can be resolved by referring to
> > the
> > > > > > previous
> > > > > > > > > > > > responses.
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > key principle is that the Catalog that has
> > been
> > > > > most
> > > > > > > > > recently
> > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye
> <
> > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> believe
> > > this
> > > > > > Flip
> > > > > > > > > could
> > > > > > > > > > be
> > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > for users. The firm I work for also
> manages a
> > > lot
> > > > > of
> > > > > > > > > > catalogs,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > submission of tasks becomes slow because of
> > > > > loading a
> > > > > > > > > number
> > > > > > > > > > of
> > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > We obtain the catalogs in the user's Flink
> > SQL
> > > > > > through
> > > > > > > > > > regular
> > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > to avoid loading all catalogs to improve
> the
> > > > speed
> > > > > of
> > > > > > > > task
> > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > questions:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. When creating a catalog with CREATE
> > > CATALOG,
> > > > > the
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > registration method is used by default.
> What
> > if
> > > > > > > > > asynchronous
> > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > failed? And how to implement asynchronous
> > > > > > registration?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. I also have the same question with
> > > Shammon’s
> > > > > > first
> > > > > > > > > > > comment.
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > CatalogStore
> > > > > > > > catalogstore
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogManager all have the same catalog
> > name,
> > > > but
> > > > > > > > > > correponding
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > different catalog instances. For example:
> > > > > catalog1->
> > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in
> the
> > > > > > > > catalogstore.
> > > > > > > > > > > Would
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > happen? Maybe we should define `catalogs`
> and
> > > > > > > > > `catalogstore`
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > ruanhang1993@gmail.com
> > > > > > > >
> > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by liu ron <ro...@gmail.com>.
Hi, Feng
Thanks for driving this FLIP, this proposal is very useful for catalog
management.
I have some small questions:
1. Regarding the CatalogStoreFactory#createCatalogStore method, do we need
to provide a default implementation?
2. If we get Catalog from CatalogStore, after initializing it, whether we
put it to Map<String, Catalog> catalogs again?
3. Regarding the options `sql.catalog.store.type` and
`sql.catalog.store.file.path`, how about renaming them to
`catalog.store.type` and `catalog.store.path`?
Best,
Ron
Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> Hi yuxia
>
> > But from the code in Proposed Changes, once we register the Catalog, we
> initialize it and open it. right?
>
> Yes, In order to avoid inconsistent semantics of the original CREATE
> CATALOG DDL, Catalog will be directly initialized in registerCatalog so
> that parameter validation can be performed.
>
> In the current design, lazy initialization is mainly reflected in
> getCatalog. If CatalogStore has already saved some catalog configurations,
> only initialization is required in getCatalog.
>
>
> Best,
> Feng
>
> On Mon, May 29, 2023 at 8:27 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> > Hi, Feng.
> > I'm trying to understanding the meaning of *lazy initialization*. If i'm
> > wrong, please correct me.
> >
> > IIUC, lazy initialization means only you need to access the catalog, then
> > you initialize it. But from the code in Proposed Changes, once we
> register
> > the Catalog,
> > we initialize it and open it. right?
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration
> > and asynchronous registration
> >
> > Hi Feng,
> >
> > Thanks for your effort! +1 for the proposal.
> >
> > One of the major changes is that current design will provide
> > Map<String,Catalog> catalogs as a snapshot instead of a cache, which
> means
> > once it has been initialized, any changes done by other sessions will not
> > affect it. Point 6 described follow-up options for further improvement.
> >
> > Best regards,
> > Jing
> >
> > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com> wrote:
> >
> > > Hi all, I would like to update you on the latest progress of the FLIP.
> > >
> > >
> > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang
> and I
> > > had an offline discussion regarding the overall solution for Flink
> > > CatalogStore. We have reached a consensus and I have updated the final
> > > solution in FLIP.
> > >
> > > Next, let me briefly describe the entire design:
> > >
> > > 1.
> > >
> > > Introduce CatalogDescriptor to store catalog configuration similar
> to
> > > TableDescriptor.
> > > 2.
> > >
> > > The two key functions of CatalogStore - void storeCatalog(String
> > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > getCatalog(String)
> > > will both use CatalogDescriptor instead of Catalog instance. This
> way,
> > > CatalogStore will only be responsible for saving and retrieving
> > catalog
> > > configurations without having to initialize catalogs.
> > > 3.
> > >
> > > The default registerCatalog(String catalogName, Catalog catalog)
> > > function in CatalogManager will be marked as deprecated.
> > > 4.
> > >
> > > A new function registerCatalog(String catalogName, CatalogDescriptor
> > > catalog) will be added to serve as the default registration function
> > for
> > > catalogs in CatalogManager.
> > > 5.
> > >
> > > Map<String,Catalog> catalogs in CataloManager will remain unchanged
> > and
> > > save initialized catalogs.This means that deletion operations from
> one
> > > session won't synchronize with other sessions.
> > > 6.
> > >
> > > To support multi-session synchronization scenarios for deletions
> later
> > > on we should make Map<String,Catalog>catalogs configurable.There may
> > be
> > > three possible situations:
> > >
> > > a.Default caching of all initialized catalogs
> > >
> > > b.Introduction of LRU cache logic which can automatically clear
> > > long-unused catalogs.
> > >
> > > c.No caching of any instances; each call to getCatalog creates a new
> > > instance.
> > >
> > >
> > > This is the document for discussion:
> > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > >
> > > This is the final proposal document:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >
> > >
> > > Thank you very much for your attention and suggestions on this FLIP. A
> > > special thanks to Hang Ruan for his suggestions on the entire design
> and
> > > organizing offline discussions.
> > >
> > > If you have any further suggestions or feedback about this FLIP please
> > feel
> > > free to share.
> > >
> > >
> > > Best,
> > >
> > > Feng
> > >
> > > On Sat, May 6, 2023 at 8:32 PM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> > >
> > > > Hi Feng,
> > > >
> > > > Thanks for improving the FLIP. It looks good to me. We could still
> > > > reconsider in the future how to provide more common built-in cache
> > > > functionality in CatalogManager so that not every CatalogSotre
> > > > implementation has to take care of it.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com>
> wrote:
> > > >
> > > > > Hi Jing,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > > There might be more such issues. I would suggest you completely
> > walk
> > > > > through the FLIP again and fix those issues
> > > > >
> > > > > I am very sorry for my carelessness and at the same time, I greatly
> > > > > appreciate your careful review. I have thoroughly checked the
> entire
> > > FLIP
> > > > > and made corrections to these issues.
> > > > >
> > > > >
> > > > > > If I am not mistaken, with the current FLIP design,
> > CatalogManager
> > > > > could work without Optional CatalogStore being configured.
> > > > >
> > > > > Yes, in the original design, CatalogStore was not necessary because
> > > > > CatalogManager used Map<String, Catalog> catalogs to store catalog
> > > > > instances.
> > > > > However, this caused inconsistency issues. Therefore, I modified
> this
> > > > part
> > > > > of the design and removed Map<String, Catalog> catalogs from
> > > > > CatalogManager.
> > > > > At the same time, InMemoryCatalog will serve as the default
> > > CatalogStore
> > > > > to save catalogs in memory and replace the functionality of
> > > > > Map<String,Catalog>catalogs.
> > > > > The previous plan that kept Map<String,Catalog>catalogs has been
> > moved
> > > to
> > > > > Rejected Alternatives.
> > > > >
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge <jing@ververica.com.invalid
> >
> > > > > wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > There are still many places contain inconsistent content, e.g.
> > > > > >
> > > > > > 1. "Asynchronous registration" is still used.
> > > > > > 2. The java comment of the method registerCatalog(String
> > catalogName,
> > > > > > Catalog catalog) in CatalogManager does not tell what the method
> is
> > > > > doing.
> > > > > >
> > > > > >
> > > > > > There might be more such issues. I would suggest you completely
> > walk
> > > > > > through the FLIP again and fix those issues.
> > > > > >
> > > > > > About the inMemoryCatalogStore, do you mean that you will build
> the
> > > > cache
> > > > > > functionality in the CatalogStore? This is a very different
> design
> > > > > concept
> > > > > > from what the current FLIP described. If I am not mistaken, with
> > the
> > > > > > current FLIP design, CatalogManager could work without Optional
> > > > > > CatalogStore being configured. That is the reason why I mentioned
> > in
> > > > the
> > > > > > last email that the example code wrt the Optional CatalogStore is
> > not
> > > > > > correct.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <ji...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Jing
> > > > > > >
> > > > > > >
> > > > > > > > There are still some NIT issues in the FLIP
> > > > > > >
> > > > > > > Thank you very much for the careful review. I have already made
> > the
> > > > > > > relevant changes.
> > > > > > >
> > > > > > >
> > > > > > > > Speaking of the conflict issues in the multi-instance
> > > scenarios, I
> > > > > am
> > > > > > > not
> > > > > > > sure if this is the intended behaviour
> > > > > > >
> > > > > > > Currently, there are conflicts in multiple scenarios with the
> > > current
> > > > > > > design. I am thinking whether we should remove 'Map<String,
> > > Catalog>'
> > > > > and
> > > > > > > make Cache the default behavior of InMemoryCatalogStore. This
> > way,
> > > > > users
> > > > > > > can implement their own CatalogStore to achieve multi-instance
> > > > > > > non-conflicting scenarios. What do you think?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > Thanks for working on the FLIP. There are still some NIT
> issues
> > > in
> > > > > the
> > > > > > > FLIP
> > > > > > > > like:
> > > > > > > >
> > > > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > > > CatalogStore
> > > > > > > > instead of Optional in the code example. It should be fine to
> > use
> > > > it
> > > > > as
> > > > > > > > pseudo code for now and update it after you submit the PR.
> > > > > > > > 2. addCatalog(...) is still used somewhere in the rejected
> > > section
> > > > > > which
> > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > >
> > > > > > > > Speaking of the conflict issues in the multi-instance
> > scenarios,
> > > I
> > > > am
> > > > > > not
> > > > > > > > sure if this is the intended behaviour. If Map<String,
> Catalog>
> > > > > > catalogs
> > > > > > > is
> > > > > > > > used as a cache, it should be invalid, once the related
> catalog
> > > has
> > > > > > been
> > > > > > > > removed from the CatalogStore by another instance. Did I miss
> > > > > > something?
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > jinfeng1995@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jing,Shammon
> > > > > > > > > Thanks for your reply.
> > > > > > > > >
> > > > > > > > > @Jing
> > > > > > > > >
> > > > > > > > > > How about persistCatalog()?
> > > > > > > > > I think this is a good function name, I have updated it in
> > the
> > > > > > > > > documentation.
> > > > > > > > >
> > > > > > > > > > Some common cache features should be implemented
> > > > > > > > > Thank you for the suggestion. If alternative 1 turns out to
> > be
> > > > more
> > > > > > > > > appropriate later, I will improve this part of the content.
> > > > > > > > >
> > > > > > > > > > As the above discussion moves forward, the option 2
> > solution
> > > > > looks
> > > > > > > more
> > > > > > > > > like a replacement of option 1
> > > > > > > > > Yes, after discussing with Shammon offline, we think that
> > > > solution
> > > > > 2
> > > > > > > > might
> > > > > > > > > be more suitable and also avoid any inconsistency issues.
> > > > > > > > >
> > > > > > > > > > There are some inconsistent descriptions in the content.
> > > Would
> > > > > you
> > > > > > > > like
> > > > > > > > > to clean them up?
> > > > > > > > > I will do my best to improve the document and appreciate
> your
> > > > > > > > suggestions.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > @Shammon
> > > > > > > > > > can you put the unselected option in `Rejected
> > Alternatives`
> > > > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > zjureel@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng
> > > > > > > > > >
> > > > > > > > > > Thanks for your update.
> > > > > > > > > >
> > > > > > > > > > I found there are two options in `Proposed Changes`, can
> > you
> > > > put
> > > > > > the
> > > > > > > > > > unselected option in `Rejected Alternatives`? I think
> this
> > > may
> > > > > help
> > > > > > > us
> > > > > > > > > > better understand your proposal
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shammon FY
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for raising this FLIP. I am still confused after
> > > > > > completely
> > > > > > > > > > reading
> > > > > > > > > > > the thread with following questions:
> > > > > > > > > > >
> > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> addCatalog()
> > > have
> > > > > no
> > > > > > > big
> > > > > > > > > > > difference based on their names. One of them is
> > responsible
> > > > for
> > > > > > > data
> > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > 2. As you mentioned that Map<String, Catalog> catalogs
> is
> > > > used
> > > > > > as a
> > > > > > > > > cache
> > > > > > > > > > > and catalogStore is used for data persistence. I would
> > > > suggest
> > > > > > > > > describing
> > > > > > > > > > > their purpose conceptually and clearly in the FLIP.
> Some
> > > > common
> > > > > > > cache
> > > > > > > > > > > features should be implemented, i.e. data in the cache
> > and
> > > > the
> > > > > > > store
> > > > > > > > > > should
> > > > > > > > > > > be consistent. Same Catalog instance should be found in
> > the
> > > > > store
> > > > > > > and
> > > > > > > > > in
> > > > > > > > > > > the cache(either it has been initialized or it will be
> > lazy
> > > > > > > > > initialized)
> > > > > > > > > > > for the same catalog name. The consistency will be
> taken
> > > care
> > > > > of
> > > > > > > > while
> > > > > > > > > > > updating the catalog.
> > > > > > > > > > > 3. As the above discussion moves forward, the option 2
> > > > solution
> > > > > > > looks
> > > > > > > > > > more
> > > > > > > > > > > like a replacement of option 1, because, afaiu, issues
> > > > > mentioned
> > > > > > > > > > > previously with option 1 are not solved yet. Do you
> still
> > > > want
> > > > > to
> > > > > > > > > propose
> > > > > > > > > > > both options and ask for suggestions for both of them?
> > > > > > > > > > > 4. After you updated the FLIP, there are some
> > inconsistent
> > > > > > > > descriptions
> > > > > > > > > > in
> > > > > > > > > > > the content. Would you like to clean them up? Thanks!
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > jinfeng1995@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > hi Shammon
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you for your response, and I completely agree
> > with
> > > > your
> > > > > > > point
> > > > > > > > > of
> > > > > > > > > > > > view.
> > > > > > > > > > > > Initially, I may have over complicated the whole
> issue.
> > > > First
> > > > > > and
> > > > > > > > > > > foremost,
> > > > > > > > > > > > we need to consider the persistence of the Catalog's
> > > > > > > Configuration.
> > > > > > > > > > > > If we only need to provide persistence for Catalog
> > > > > > Configuration,
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > add a toConfiguration method to the Catalog
> interface.
> > > > > > > > > > > > This method can convert a Catalog instance to a
> > > Map<String,
> > > > > > > String>
> > > > > > > > > > > > properties, and the default implementation will throw
> > an
> > > > > > > exception.
> > > > > > > > > > > >
> > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > /**
> > > > > > > > > > > > * Returns a map containing the properties of the
> > > catalog
> > > > > > > object.
> > > > > > > > > > > > *
> > > > > > > > > > > > * @return a map containing the properties of the
> > > catalog
> > > > > > > object
> > > > > > > > > > > > * @throws UnsupportedOperationException if the
> > > > > implementing
> > > > > > > > class
> > > > > > > > > > does
> > > > > > > > > > > > not override
> > > > > > > > > > > > * the default implementation of this method
> > > > > > > > > > > > */
> > > > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > > > throw new UnsupportedOperationException("Please
> > > > implement
> > > > > > > > > > > toProperties
> > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > }
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. If the user has configured a CatalogStore, the
> > > > > > > > toConfiguration()
> > > > > > > > > > > method
> > > > > > > > > > > > will be called when registering a Catalog instance
> with
> > > > > > > > > > > > registerCatalog(String catalogName, Catalog catalog).
> > The
> > > > > > Catalog
> > > > > > > > > > > instance
> > > > > > > > > > > > will be converted to a Map<String, String> properties
> > and
> > > > > saved
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > CatalogStore. If some Catalog instances do not
> > implement
> > > > this
> > > > > > > > method,
> > > > > > > > > > an
> > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. If the user does not use a CatalogStore, the
> > > > > > > toConfiguration()
> > > > > > > > > > method
> > > > > > > > > > > > will not be called, ensuring consistency with the
> > > original
> > > > > > > process.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. Saving both the Map<String, Catalog> catalogs and
> > the
> > > > > > > > CatalogStore
> > > > > > > > > > at
> > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > we can start from the Catalog itself and provide a
> > > > dedicated
> > > > > > > > Catalog
> > > > > > > > > > > > implementation for lazy loading, such as
> > > > ConfigurationCatalog
> > > > > > > > > > > >
> > > > > > > > > > > > public class ConfigurationCatalog implements Catalog
> {
> > > > > > > > > > > > ConfigurationCatalog(Map<String, String> properties)
> {
> > > > > > > > > > > > }
> > > > > > > > > > > > }
> > > > > > > > > > > >
> > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > zjureel@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have no questions about the functionality of
> > > > > > `CatalogStore`,
> > > > > > > > but
> > > > > > > > > > the
> > > > > > > > > > > > > different operations of multiple `registerCatalog`
> or
> > > > > > > > > `storeCatalog`
> > > > > > > > > > in
> > > > > > > > > > > > > `CatalogManager`. The implementation in Trino is
> also
> > > the
> > > > > > same,
> > > > > > > > the
> > > > > > > > > > > > > `CatalogManager` in trino only has one
> > `createCatalog`,
> > > > > which
> > > > > > > > will
> > > > > > > > > > save
> > > > > > > > > > > > the
> > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think we don't need to add another
> > `registerCatalog`
> > > or
> > > > > > > > > > `addCatalog`
> > > > > > > > > > > to
> > > > > > > > > > > > > save a catalog in `Map<String, Catalog> catalogs`
> or
> > > > > > > > `CatalogStore
> > > > > > > > > > > > > catalogStore`. As you mentioned before, the
> > > `Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How about
> > > > saving
> > > > > > the
> > > > > > > > > > catalog
> > > > > > > > > > > in
> > > > > > > > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore
> > > > > > catalogStore`
> > > > > > > > > > > together
> > > > > > > > > > > > > when it is registered or added in `CatalogManager`?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > jinfeng1995@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for your reply. I am very sorry for the
> > > > > > > > > misunderstanding
> > > > > > > > > > > > caused
> > > > > > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > I found there is a pre-discussion [1] for this
> > FLIP
> > > > > > > > > > > > > > Yes, there was indeed such a discussion before.
> > > > However,
> > > > > > in
> > > > > > > > > > > designing
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > CatalogManager
> > > > > > > itself
> > > > > > > > > > > doesn't
> > > > > > > > > > > > > > need to change much. *We cannot only persist
> > Catalog
> > > > > > > instances
> > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > so exposing only registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > catalog)
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > not be enough to save Catalogs, because in the
> end
> > we
> > > > > still
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > save
> > > > > > > > > > > > > > the configurations corresponding to the Catalog
> > > > > instances.
> > > > > > > > > > > Therefore,
> > > > > > > > > > > > I
> > > > > > > > > > > > > > decided to introduce the CatalogStore interface
> for
> > > > > > > > configuration
> > > > > > > > > > > > > > persistence. Regarding this part, I also took
> > > > inspiration
> > > > > > > from
> > > > > > > > > > > Trino's
> > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1: The mechanism of handling catalog with the
> > same
> > > > name
> > > > > > > > looks a
> > > > > > > > > > > > little
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > Thank you for the suggestion. I will provide a
> > > detailed
> > > > > > > > > description
> > > > > > > > > > > and
> > > > > > > > > > > > > > code examples for this part, and add it to the
> > FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still
> looks
> > > > > confused
> > > > > > > to
> > > > > > > > > me.
> > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > this method is for storing catalog to
> CatalogStore,
> > > how
> > > > > > about
> > > > > > > > > > > renaming
> > > > > > > > > > > > it
> > > > > > > > > > > > > > to `storeCatalog`? It's very personal opinion,
> you
> > > can
> > > > > > decide
> > > > > > > > to
> > > > > > > > > > take
> > > > > > > > > > > > it
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > StoreCatalog looks more intuitive to me. I don't
> > see
> > > > any
> > > > > > > > problem
> > > > > > > > > > with
> > > > > > > > > > > > it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which method
> will
> > > be
> > > > > > > called?
> > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> > > > whether
> > > > > > user
> > > > > > > > can
> > > > > > > > > > > add a
> > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > For CREATE CATALOG, my original design was to add
> > it
> > > > > > directly
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > CatalogStore, but this would disrupt the existing
> > > > logic.
> > > > > > > > > > Therefore, I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > we can do both: save the configuration to the
> > > > > CatalogStore
> > > > > > > and
> > > > > > > > > > > > initialize
> > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. Is it really neccessary to provide a default
> > > > > > > implmentation
> > > > > > > > > for
> > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > I think it is necessary, otherwise we would need
> to
> > > > > > introduce
> > > > > > > > an
> > > > > > > > > > > > > additional
> > > > > > > > > > > > > > Map to store the configuration for lazy loading.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > > > > I don't think registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > can
> > > > > > > > > > > > be
> > > > > > > > > > > > > > made into an asynchronous interface because
> Catalog
> > > is
> > > > > > > already
> > > > > > > > an
> > > > > > > > > > > > > instance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It looks more like lazy initialization for
> > catalog
> > > > than
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > Indeed, my description was inaccurate. It should
> be
> > > > lazy
> > > > > > > > > > registration
> > > > > > > > > > > > > > instead of asynchronous registration. I have
> > already
> > > > > > updated
> > > > > > > > the
> > > > > > > > > > > title
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > > > zjureel@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think if there is a `registerCatalog` method
> in
> > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > confuse users whether a method named
> `addCatalog`
> > > or
> > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > And as you mentioned, the memory catalog is a
> > > > `cache`,
> > > > > I
> > > > > > > > think
> > > > > > > > > > the
> > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I found there is a pre-discussion [1] for this
> > > FLIP.
> > > > > > Please
> > > > > > > > > > correct
> > > > > > > > > > > > me
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > discussion
> > > is
> > > > > to
> > > > > > > use
> > > > > > > > > > > > > > > `CatalogManager` as an interface and implement
> it
> > > for
> > > > > > > > different
> > > > > > > > > > > > stores
> > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think there is a gap between the current FLIP
> > > > design
> > > > > > and
> > > > > > > > that
> > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > What about the proposal of the discussion in
> > thread
> > > > > [1] ?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with the
> > > same
> > > > > name
> > > > > > > > > looks a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > complex to me. I think it'll be better to
> > explain
> > > > it
> > > > > in
> > > > > > > the
> > > > > > > > > > java
> > > > > > > > > > > > doc
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > these methods and give a brief example in
> this
> > > > FLIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still
> > looks
> > > > > > confused
> > > > > > > > to
> > > > > > > > > > me.
> > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > this method is for storing catalog to
> > > CatalogStore,
> > > > > how
> > > > > > > > about
> > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> opinion,
> > > you
> > > > > can
> > > > > > > > decide
> > > > > > > > > > to
> > > > > > > > > > > > take
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which method
> > > will
> > > > be
> > > > > > > > called?
> > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > wondering
> > > > > > whether
> > > > > > > > user
> > > > > > > > > > can
> > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3: Is it really neccessary to provide a
> default
> > > > > > > > implmentation
> > > > > > > > > > for
> > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 4: About asynchronous registration for
> catalog.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> CATALOG,
> > > the
> > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > > > If asynchronous registration is the default
> > > > behavior,
> > > > > > it
> > > > > > > > > there
> > > > > > > > > > > any
> > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > that user can switch to synchronous
> > registration
> > > > just
> > > > > > > like
> > > > > > > > > > > before?
> > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > `registerCatalog`
> > > > > be
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > IIUC, in asynchronous registration, it may
> well
> > > > that
> > > > > > > CREATE
> > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > executes successfully, but then the following
> > > > CREATE
> > > > > > > TABLE
> > > > > > > > > > > > statement
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > I think it's a break change which should be
> > > > > highlighted
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > BTW, by saying asynchronous registration, I
> > would
> > > > > like
> > > > > > to
> > > > > > > > > > expect
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > will be an executor to open or register
> catalog
> > > in
> > > > > the
> > > > > > > > > > > background,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > "the Catalog instance will be initialized if
> it
> > > has
> > > > > not
> > > > > > > > been
> > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > yet. If the initialization process fails,
> these
> > > > > > > statements
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > It looks more like lazy initialization for
> > > catalog
> > > > > than
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> persistence
> > > of
> > > > > > > Catalog
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi everyone, Thank you all for your interest
> in
> > > > this
> > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > How to handle a catalog with the same name
> > that
> > > > > > exists
> > > > > > > > for
> > > > > > > > > > both
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I believe this is a crucial point. Based on
> my
> > > > > current
> > > > > > > > > personal
> > > > > > > > > > > > > > > > understanding, the Map<String, Catalog>
> > catalogs
> > > > will
> > > > > > > serve
> > > > > > > > > as
> > > > > > > > > > a
> > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > for instantiated catalogs and have the
> highest
> > > > > > priority.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are three methods that can affect the
> > > > > Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. registerCatalog(String catalogName,
> Catalog
> > > > > catalog)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This method puts the catalog instance into
> the
> > > > > > > Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This method removes the catalog instance
> > > > > corresponding
> > > > > > to
> > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This method first retrieves the corresponding
> > > > catalog
> > > > > > > > > instance
> > > > > > > > > > > from
> > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If the
> > catalog
> > > > > does
> > > > > > > not
> > > > > > > > > > exist,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > retrieves the corresponding configuration
> from
> > > the
> > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > initializes it, and puts the initialized
> > Catalog
> > > > > > instance
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The following two methods only modify the
> > > > > configuration
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. addCatalog(String catalogName, Map<String,
> > > > String>
> > > > > > > > > > properties)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This method saves the properties to the
> > > > catalogStore
> > > > > > and
> > > > > > > > > checks
> > > > > > > > > > > > > > > > whether there is a catalogName with the same
> > > name.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > This method removes the specified
> configuration
> > > of
> > > > > the
> > > > > > > > > > specified
> > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The following are possible conflict
> scenarios:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. When the corresponding catalogName already
> > > > exists
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore but not in the Map<String,
> > Catalog>,
> > > > the
> > > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > > catalog)
> > > > > > > method
> > > > > > > > > can
> > > > > > > > > > > > > > > > succeed and be directly saved to the
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. When the corresponding catalogName already
> > > > exists
> > > > > in
> > > > > > > > both
> > > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore and the Map<String, Catalog>,
> the
> > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > catalogName, Catalog catalog) method will
> fail.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. When the corresponding catalogName already
> > > > exists
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > > > > Map<String, Catalog>, the addCatalog(String
> > > > > > catalogName,
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > String> properties) method can directly save
> > the
> > > > > > > properties
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > > > catalogName)
> > > > > > > method
> > > > > > > > > > will
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > use the new properties for initialization
> > because
> > > > the
> > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > catalog instance already exists in catalogs
> and
> > > > will
> > > > > be
> > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > Therefore, using the unregisterCatalog(String
> > > > > > > catalogName)
> > > > > > > > > > method
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > remove the instance corresponding to the
> > original
> > > > > > > > catalogName
> > > > > > > > > > is
> > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > `Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > Map<String,
> > > > > > String>
> > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This could potentially lead to confusion. I
> > > suggest
> > > > > > > > changing
> > > > > > > > > > the
> > > > > > > > > > > > > > > > method name, perhaps to addCatalog(String
> > > > > catalogName,
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > String> properties), as previously mentioned
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > add `registerCatalog(String
> > catalogName,Catalog
> > > > > > > catalog,
> > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Since a catalog is already an instance,
> adding
> > > the
> > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > parameter to the registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > method may not necessarily result in lazy
> > > > > > initialization.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think encryption is necessary, but perhaps
> > the
> > > > > > > encryption
> > > > > > > > > > logic
> > > > > > > > > > > > > > > > should be implemented in the CatalogStore
> when
> > > the
> > > > > data
> > > > > > > is
> > > > > > > > > > > actually
> > > > > > > > > > > > > > > > saved. For instance, the FileCatalogStore
> could
> > > > > encrypt
> > > > > > > the
> > > > > > > > > > data
> > > > > > > > > > > > when
> > > > > > > > > > > > > > > > it is saved to a file, while the
> > > MemoryCatalogStore
> > > > > > does
> > > > > > > > not
> > > > > > > > > > > > require
> > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Do we really need the
> `MemoryCatalogStore`?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > MemoryCatalogStore
> > > > > as
> > > > > > > the
> > > > > > > > > > > default
> > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > configurations
> > > > in
> > > > > > > > memory.
> > > > > > > > > > > > > > > > Otherwise, if we want to implement
> asynchronous
> > > > > loading
> > > > > > > of
> > > > > > > > > > > Catalog,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is also a critical concern. When
> > executing
> > > > DDL,
> > > > > > > DQL,
> > > > > > > > or
> > > > > > > > > > DML
> > > > > > > > > > > > > > > > statements that reference a specific Catalog,
> > the
> > > > > > Catalog
> > > > > > > > > > > instance
> > > > > > > > > > > > > > > > will be initialized if it has not been
> > > initialized
> > > > > yet.
> > > > > > > If
> > > > > > > > > the
> > > > > > > > > > > > > > > > initialization process fails, these
> statements
> > > will
> > > > > not
> > > > > > > be
> > > > > > > > > > > executed
> > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs and
> > > > > > CatalogStore
> > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the CatalogManager all have the same catalog
> > > name,
> > > > > but
> > > > > > > > > > > correponding
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This issue can be resolved by referring to
> the
> > > > > previous
> > > > > > > > > > > responses.
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > key principle is that the Catalog that has
> been
> > > > most
> > > > > > > > recently
> > > > > > > > > > > used
> > > > > > > > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do believe
> > this
> > > > > Flip
> > > > > > > > could
> > > > > > > > > be
> > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > for users. The firm I work for also manages a
> > lot
> > > > of
> > > > > > > > > catalogs,
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > submission of tasks becomes slow because of
> > > > loading a
> > > > > > > > number
> > > > > > > > > of
> > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > We obtain the catalogs in the user's Flink
> SQL
> > > > > through
> > > > > > > > > regular
> > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > to avoid loading all catalogs to improve the
> > > speed
> > > > of
> > > > > > > task
> > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > After reading flip-295, I have some
> questions:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. When creating a catalog with CREATE
> > CATALOG,
> > > > the
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration method is used by default. What
> if
> > > > > > > > asynchronous
> > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > failed? And how to implement asynchronous
> > > > > registration?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. I also have the same question with
> > Shammon’s
> > > > > first
> > > > > > > > > > comment.
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> CatalogStore
> > > > > > > catalogstore
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogManager all have the same catalog
> name,
> > > but
> > > > > > > > > correponding
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > different catalog instances. For example:
> > > > catalog1->
> > > > > > > > > > jdbcCatalog
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the
> > > > > > > catalogstore.
> > > > > > > > > > Would
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > happen? Maybe we should define `catalogs` and
> > > > > > > > `catalogstore`
> > > > > > > > > > more
> > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > ruanhang1993@gmail.com
> > > > > > >
> > > > > > > > 写道：
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi yuxia
> But from the code in Proposed Changes, once we register the Catalog, we
initialize it and open it. right?
Yes, In order to avoid inconsistent semantics of the original CREATE
CATALOG DDL, Catalog will be directly initialized in registerCatalog so
that parameter validation can be performed.
In the current design, lazy initialization is mainly reflected in
getCatalog. If CatalogStore has already saved some catalog configurations,
only initialization is required in getCatalog.
Best,
Feng
On Mon, May 29, 2023 at 8:27 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Hi, Feng.
> I'm trying to understanding the meaning of *lazy initialization*. If i'm
> wrong, please correct me.
>
> IIUC, lazy initialization means only you need to access the catalog, then
> you initialize it. But from the code in Proposed Changes, once we register
> the Catalog,
> we initialize it and open it. right?
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration
> and asynchronous registration
>
> Hi Feng,
>
> Thanks for your effort! +1 for the proposal.
>
> One of the major changes is that current design will provide
> Map<String,Catalog> catalogs as a snapshot instead of a cache, which means
> once it has been initialized, any changes done by other sessions will not
> affect it. Point 6 described follow-up options for further improvement.
>
> Best regards,
> Jing
>
> On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com> wrote:
>
> > Hi all, I would like to update you on the latest progress of the FLIP.
> >
> >
> > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang and I
> > had an offline discussion regarding the overall solution for Flink
> > CatalogStore. We have reached a consensus and I have updated the final
> > solution in FLIP.
> >
> > Next, let me briefly describe the entire design:
> >
> > 1.
> >
> > Introduce CatalogDescriptor to store catalog configuration similar to
> > TableDescriptor.
> > 2.
> >
> > The two key functions of CatalogStore - void storeCatalog(String
> > catalogName, CatalogDescriptor) and CatalogDescriptor
> getCatalog(String)
> > will both use CatalogDescriptor instead of Catalog instance. This way,
> > CatalogStore will only be responsible for saving and retrieving
> catalog
> > configurations without having to initialize catalogs.
> > 3.
> >
> > The default registerCatalog(String catalogName, Catalog catalog)
> > function in CatalogManager will be marked as deprecated.
> > 4.
> >
> > A new function registerCatalog(String catalogName, CatalogDescriptor
> > catalog) will be added to serve as the default registration function
> for
> > catalogs in CatalogManager.
> > 5.
> >
> > Map<String,Catalog> catalogs in CataloManager will remain unchanged
> and
> > save initialized catalogs.This means that deletion operations from one
> > session won't synchronize with other sessions.
> > 6.
> >
> > To support multi-session synchronization scenarios for deletions later
> > on we should make Map<String,Catalog>catalogs configurable.There may
> be
> > three possible situations:
> >
> > a.Default caching of all initialized catalogs
> >
> > b.Introduction of LRU cache logic which can automatically clear
> > long-unused catalogs.
> >
> > c.No caching of any instances; each call to getCatalog creates a new
> > instance.
> >
> >
> > This is the document for discussion:
> >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> >
> > This is the final proposal document:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >
> >
> > Thank you very much for your attention and suggestions on this FLIP. A
> > special thanks to Hang Ruan for his suggestions on the entire design and
> > organizing offline discussions.
> >
> > If you have any further suggestions or feedback about this FLIP please
> feel
> > free to share.
> >
> >
> > Best,
> >
> > Feng
> >
> > On Sat, May 6, 2023 at 8:32 PM Jing Ge <ji...@ververica.com.invalid>
> wrote:
> >
> > > Hi Feng,
> > >
> > > Thanks for improving the FLIP. It looks good to me. We could still
> > > reconsider in the future how to provide more common built-in cache
> > > functionality in CatalogManager so that not every CatalogSotre
> > > implementation has to take care of it.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com> wrote:
> > >
> > > > Hi Jing,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > > There might be more such issues. I would suggest you completely
> walk
> > > > through the FLIP again and fix those issues
> > > >
> > > > I am very sorry for my carelessness and at the same time, I greatly
> > > > appreciate your careful review. I have thoroughly checked the entire
> > FLIP
> > > > and made corrections to these issues.
> > > >
> > > >
> > > > > If I am not mistaken, with the current FLIP design,
> CatalogManager
> > > > could work without Optional CatalogStore being configured.
> > > >
> > > > Yes, in the original design, CatalogStore was not necessary because
> > > > CatalogManager used Map<String, Catalog> catalogs to store catalog
> > > > instances.
> > > > However, this caused inconsistency issues. Therefore, I modified this
> > > part
> > > > of the design and removed Map<String, Catalog> catalogs from
> > > > CatalogManager.
> > > > At the same time, InMemoryCatalog will serve as the default
> > CatalogStore
> > > > to save catalogs in memory and replace the functionality of
> > > > Map<String,Catalog>catalogs.
> > > > The previous plan that kept Map<String,Catalog>catalogs has been
> moved
> > to
> > > > Rejected Alternatives.
> > > >
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi Feng,
> > > > >
> > > > > There are still many places contain inconsistent content, e.g.
> > > > >
> > > > > 1. "Asynchronous registration" is still used.
> > > > > 2. The java comment of the method registerCatalog(String
> catalogName,
> > > > > Catalog catalog) in CatalogManager does not tell what the method is
> > > > doing.
> > > > >
> > > > >
> > > > > There might be more such issues. I would suggest you completely
> walk
> > > > > through the FLIP again and fix those issues.
> > > > >
> > > > > About the inMemoryCatalogStore, do you mean that you will build the
> > > cache
> > > > > functionality in the CatalogStore? This is a very different design
> > > > concept
> > > > > from what the current FLIP described. If I am not mistaken, with
> the
> > > > > current FLIP design, CatalogManager could work without Optional
> > > > > CatalogStore being configured. That is the reason why I mentioned
> in
> > > the
> > > > > last email that the example code wrt the Optional CatalogStore is
> not
> > > > > correct.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Jing
> > > > > >
> > > > > >
> > > > > > > There are still some NIT issues in the FLIP
> > > > > >
> > > > > > Thank you very much for the careful review. I have already made
> the
> > > > > > relevant changes.
> > > > > >
> > > > > >
> > > > > > > Speaking of the conflict issues in the multi-instance
> > scenarios, I
> > > > am
> > > > > > not
> > > > > > sure if this is the intended behaviour
> > > > > >
> > > > > > Currently, there are conflicts in multiple scenarios with the
> > current
> > > > > > design. I am thinking whether we should remove 'Map<String,
> > Catalog>'
> > > > and
> > > > > > make Cache the default behavior of InMemoryCatalogStore. This
> way,
> > > > users
> > > > > > can implement their own CatalogStore to achieve multi-instance
> > > > > > non-conflicting scenarios. What do you think?
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > Thanks for working on the FLIP. There are still some NIT issues
> > in
> > > > the
> > > > > > FLIP
> > > > > > > like:
> > > > > > >
> > > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > > CatalogStore
> > > > > > > instead of Optional in the code example. It should be fine to
> use
> > > it
> > > > as
> > > > > > > pseudo code for now and update it after you submit the PR.
> > > > > > > 2. addCatalog(...) is still used somewhere in the rejected
> > section
> > > > > which
> > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > >
> > > > > > > Speaking of the conflict issues in the multi-instance
> scenarios,
> > I
> > > am
> > > > > not
> > > > > > > sure if this is the intended behaviour. If Map<String, Catalog>
> > > > > catalogs
> > > > > > is
> > > > > > > used as a cache, it should be invalid, once the related catalog
> > has
> > > > > been
> > > > > > > removed from the CatalogStore by another instance. Did I miss
> > > > > something?
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> jinfeng1995@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jing,Shammon
> > > > > > > > Thanks for your reply.
> > > > > > > >
> > > > > > > > @Jing
> > > > > > > >
> > > > > > > > > How about persistCatalog()?
> > > > > > > > I think this is a good function name, I have updated it in
> the
> > > > > > > > documentation.
> > > > > > > >
> > > > > > > > > Some common cache features should be implemented
> > > > > > > > Thank you for the suggestion. If alternative 1 turns out to
> be
> > > more
> > > > > > > > appropriate later, I will improve this part of the content.
> > > > > > > >
> > > > > > > > > As the above discussion moves forward, the option 2
> solution
> > > > looks
> > > > > > more
> > > > > > > > like a replacement of option 1
> > > > > > > > Yes, after discussing with Shammon offline, we think that
> > > solution
> > > > 2
> > > > > > > might
> > > > > > > > be more suitable and also avoid any inconsistency issues.
> > > > > > > >
> > > > > > > > > There are some inconsistent descriptions in the content.
> > Would
> > > > you
> > > > > > > like
> > > > > > > > to clean them up?
> > > > > > > > I will do my best to improve the document and appreciate your
> > > > > > > suggestions.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > @Shammon
> > > > > > > > > can you put the unselected option in `Rejected
> Alternatives`
> > > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Feng
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> zjureel@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng
> > > > > > > > >
> > > > > > > > > Thanks for your update.
> > > > > > > > >
> > > > > > > > > I found there are two options in `Proposed Changes`, can
> you
> > > put
> > > > > the
> > > > > > > > > unselected option in `Rejected Alternatives`? I think this
> > may
> > > > help
> > > > > > us
> > > > > > > > > better understand your proposal
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shammon FY
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for raising this FLIP. I am still confused after
> > > > > completely
> > > > > > > > > reading
> > > > > > > > > > the thread with following questions:
> > > > > > > > > >
> > > > > > > > > > 1. Naming confusion - registerCatalog() and addCatalog()
> > have
> > > > no
> > > > > > big
> > > > > > > > > > difference based on their names. One of them is
> responsible
> > > for
> > > > > > data
> > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > 2. As you mentioned that Map<String, Catalog> catalogs is
> > > used
> > > > > as a
> > > > > > > > cache
> > > > > > > > > > and catalogStore is used for data persistence. I would
> > > suggest
> > > > > > > > describing
> > > > > > > > > > their purpose conceptually and clearly in the FLIP. Some
> > > common
> > > > > > cache
> > > > > > > > > > features should be implemented, i.e. data in the cache
> and
> > > the
> > > > > > store
> > > > > > > > > should
> > > > > > > > > > be consistent. Same Catalog instance should be found in
> the
> > > > store
> > > > > > and
> > > > > > > > in
> > > > > > > > > > the cache(either it has been initialized or it will be
> lazy
> > > > > > > > initialized)
> > > > > > > > > > for the same catalog name. The consistency will be taken
> > care
> > > > of
> > > > > > > while
> > > > > > > > > > updating the catalog.
> > > > > > > > > > 3. As the above discussion moves forward, the option 2
> > > solution
> > > > > > looks
> > > > > > > > > more
> > > > > > > > > > like a replacement of option 1, because, afaiu, issues
> > > > mentioned
> > > > > > > > > > previously with option 1 are not solved yet. Do you still
> > > want
> > > > to
> > > > > > > > propose
> > > > > > > > > > both options and ask for suggestions for both of them?
> > > > > > > > > > 4. After you updated the FLIP, there are some
> inconsistent
> > > > > > > descriptions
> > > > > > > > > in
> > > > > > > > > > the content. Would you like to clean them up? Thanks!
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > jinfeng1995@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > hi Shammon
> > > > > > > > > > >
> > > > > > > > > > > Thank you for your response, and I completely agree
> with
> > > your
> > > > > > point
> > > > > > > > of
> > > > > > > > > > > view.
> > > > > > > > > > > Initially, I may have over complicated the whole issue.
> > > First
> > > > > and
> > > > > > > > > > foremost,
> > > > > > > > > > > we need to consider the persistence of the Catalog's
> > > > > > Configuration.
> > > > > > > > > > > If we only need to provide persistence for Catalog
> > > > > Configuration,
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > add a toConfiguration method to the Catalog interface.
> > > > > > > > > > > This method can convert a Catalog instance to a
> > Map<String,
> > > > > > String>
> > > > > > > > > > > properties, and the default implementation will throw
> an
> > > > > > exception.
> > > > > > > > > > >
> > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > /**
> > > > > > > > > > > * Returns a map containing the properties of the
> > catalog
> > > > > > object.
> > > > > > > > > > > *
> > > > > > > > > > > * @return a map containing the properties of the
> > catalog
> > > > > > object
> > > > > > > > > > > * @throws UnsupportedOperationException if the
> > > > implementing
> > > > > > > class
> > > > > > > > > does
> > > > > > > > > > > not override
> > > > > > > > > > > * the default implementation of this method
> > > > > > > > > > > */
> > > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > > throw new UnsupportedOperationException("Please
> > > implement
> > > > > > > > > > toProperties
> > > > > > > > > > > for this catalog");
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > >
> > > > > > > > > > > 1. If the user has configured a CatalogStore, the
> > > > > > > toConfiguration()
> > > > > > > > > > method
> > > > > > > > > > > will be called when registering a Catalog instance with
> > > > > > > > > > > registerCatalog(String catalogName, Catalog catalog).
> The
> > > > > Catalog
> > > > > > > > > > instance
> > > > > > > > > > > will be converted to a Map<String, String> properties
> and
> > > > saved
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > CatalogStore. If some Catalog instances do not
> implement
> > > this
> > > > > > > method,
> > > > > > > > > an
> > > > > > > > > > > exception will be thrown.
> > > > > > > > > > >
> > > > > > > > > > > 2. If the user does not use a CatalogStore, the
> > > > > > toConfiguration()
> > > > > > > > > method
> > > > > > > > > > > will not be called, ensuring consistency with the
> > original
> > > > > > process.
> > > > > > > > > > >
> > > > > > > > > > > 3. Saving both the Map<String, Catalog> catalogs and
> the
> > > > > > > CatalogStore
> > > > > > > > > at
> > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > we can start from the Catalog itself and provide a
> > > dedicated
> > > > > > > Catalog
> > > > > > > > > > > implementation for lazy loading, such as
> > > ConfigurationCatalog
> > > > > > > > > > >
> > > > > > > > > > > public class ConfigurationCatalog implements Catalog {
> > > > > > > > > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > > > > > > > > }
> > > > > > > > > > > }
> > > > > > > > > > >
> > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > zjureel@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > >
> > > > > > > > > > > > I have no questions about the functionality of
> > > > > `CatalogStore`,
> > > > > > > but
> > > > > > > > > the
> > > > > > > > > > > > different operations of multiple `registerCatalog` or
> > > > > > > > `storeCatalog`
> > > > > > > > > in
> > > > > > > > > > > > `CatalogManager`. The implementation in Trino is also
> > the
> > > > > same,
> > > > > > > the
> > > > > > > > > > > > `CatalogManager` in trino only has one
> `createCatalog`,
> > > > which
> > > > > > > will
> > > > > > > > > save
> > > > > > > > > > > the
> > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > >
> > > > > > > > > > > > I think we don't need to add another
> `registerCatalog`
> > or
> > > > > > > > > `addCatalog`
> > > > > > > > > > to
> > > > > > > > > > > > save a catalog in `Map<String, Catalog> catalogs` or
> > > > > > > `CatalogStore
> > > > > > > > > > > > catalogStore`. As you mentioned before, the
> > `Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How about
> > > saving
> > > > > the
> > > > > > > > > catalog
> > > > > > > > > > in
> > > > > > > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore
> > > > > catalogStore`
> > > > > > > > > > together
> > > > > > > > > > > > when it is registered or added in `CatalogManager`?
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Shammon FY
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > jinfeng1995@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thank you for your reply. I am very sorry for the
> > > > > > > > misunderstanding
> > > > > > > > > > > caused
> > > > > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > I found there is a pre-discussion [1] for this
> FLIP
> > > > > > > > > > > > > Yes, there was indeed such a discussion before.
> > > However,
> > > > > in
> > > > > > > > > > designing
> > > > > > > > > > > > the
> > > > > > > > > > > > > whole solution, I found that the logic of
> > > CatalogManager
> > > > > > itself
> > > > > > > > > > doesn't
> > > > > > > > > > > > > need to change much. *We cannot only persist
> Catalog
> > > > > > instances
> > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > so exposing only registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > > might
> > > > > > > > > > > > > not be enough to save Catalogs, because in the end
> we
> > > > still
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > > save
> > > > > > > > > > > > > the configurations corresponding to the Catalog
> > > > instances.
> > > > > > > > > > Therefore,
> > > > > > > > > > > I
> > > > > > > > > > > > > decided to introduce the CatalogStore interface for
> > > > > > > configuration
> > > > > > > > > > > > > persistence. Regarding this part, I also took
> > > inspiration
> > > > > > from
> > > > > > > > > > Trino's
> > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 1: The mechanism of handling catalog with the
> same
> > > name
> > > > > > > looks a
> > > > > > > > > > > little
> > > > > > > > > > > > of
> > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > Thank you for the suggestion. I will provide a
> > detailed
> > > > > > > > description
> > > > > > > > > > and
> > > > > > > > > > > > > code examples for this part, and add it to the
> FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> > > > confused
> > > > > > to
> > > > > > > > me.
> > > > > > > > > > > IIUC,
> > > > > > > > > > > > > this method is for storing catalog to CatalogStore,
> > how
> > > > > about
> > > > > > > > > > renaming
> > > > > > > > > > > it
> > > > > > > > > > > > > to `storeCatalog`? It's very personal opinion, you
> > can
> > > > > decide
> > > > > > > to
> > > > > > > > > take
> > > > > > > > > > > it
> > > > > > > > > > > > or
> > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > StoreCatalog looks more intuitive to me. I don't
> see
> > > any
> > > > > > > problem
> > > > > > > > > with
> > > > > > > > > > > it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which method will
> > be
> > > > > > called?
> > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> > > whether
> > > > > user
> > > > > > > can
> > > > > > > > > > add a
> > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > For CREATE CATALOG, my original design was to add
> it
> > > > > directly
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > CatalogStore, but this would disrupt the existing
> > > logic.
> > > > > > > > > Therefore, I
> > > > > > > > > > > > think
> > > > > > > > > > > > > we can do both: save the configuration to the
> > > > CatalogStore
> > > > > > and
> > > > > > > > > > > initialize
> > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. Is it really neccessary to provide a default
> > > > > > implmentation
> > > > > > > > for
> > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > I think it is necessary, otherwise we would need to
> > > > > introduce
> > > > > > > an
> > > > > > > > > > > > additional
> > > > > > > > > > > > > Map to store the configuration for lazy loading.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > > > I don't think registerCatalog(String catalogName,
> > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > can
> > > > > > > > > > > be
> > > > > > > > > > > > > made into an asynchronous interface because Catalog
> > is
> > > > > > already
> > > > > > > an
> > > > > > > > > > > > instance.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > It looks more like lazy initialization for
> catalog
> > > than
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > Indeed, my description was inaccurate. It should be
> > > lazy
> > > > > > > > > registration
> > > > > > > > > > > > > instead of asynchronous registration. I have
> already
> > > > > updated
> > > > > > > the
> > > > > > > > > > title
> > > > > > > > > > > of
> > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1].
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > > zjureel@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think if there is a `registerCatalog` method in
> > > > > > > > > `CatalogManager`,
> > > > > > > > > > > it
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > confuse users whether a method named `addCatalog`
> > or
> > > > > > > > > `storeCatalog`
> > > > > > > > > > > is
> > > > > > > > > > > > > > added.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And as you mentioned, the memory catalog is a
> > > `cache`,
> > > > I
> > > > > > > think
> > > > > > > > > the
> > > > > > > > > > > > > concept
> > > > > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I found there is a pre-discussion [1] for this
> > FLIP.
> > > > > Please
> > > > > > > > > correct
> > > > > > > > > > > me
> > > > > > > > > > > > if
> > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> discussion
> > is
> > > > to
> > > > > > use
> > > > > > > > > > > > > > `CatalogManager` as an interface and implement it
> > for
> > > > > > > different
> > > > > > > > > > > stores
> > > > > > > > > > > > > such
> > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think there is a gap between the current FLIP
> > > design
> > > > > and
> > > > > > > that
> > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > What about the proposal of the discussion in
> thread
> > > > [1] ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > >
> > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > 1: The mechanism of handling catalog with the
> > same
> > > > name
> > > > > > > > looks a
> > > > > > > > > > > > little
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > complex to me. I think it'll be better to
> explain
> > > it
> > > > in
> > > > > > the
> > > > > > > > > java
> > > > > > > > > > > doc
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > these methods and give a brief example in this
> > > FLIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still
> looks
> > > > > confused
> > > > > > > to
> > > > > > > > > me.
> > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > this method is for storing catalog to
> > CatalogStore,
> > > > how
> > > > > > > about
> > > > > > > > > > > > renaming
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > to `storeCatalog`? It's very personal opinion,
> > you
> > > > can
> > > > > > > decide
> > > > > > > > > to
> > > > > > > > > > > take
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which method
> > will
> > > be
> > > > > > > called?
> > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> wondering
> > > > > whether
> > > > > > > user
> > > > > > > > > can
> > > > > > > > > > > > add a
> > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3: Is it really neccessary to provide a default
> > > > > > > implmentation
> > > > > > > > > for
> > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > When creating a catalog with CREATE CATALOG,
> > the
> > > > > > > > asynchronous
> > > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > > If asynchronous registration is the default
> > > behavior,
> > > > > it
> > > > > > > > there
> > > > > > > > > > any
> > > > > > > > > > > > way
> > > > > > > > > > > > > > > that user can switch to synchronous
> registration
> > > just
> > > > > > like
> > > > > > > > > > before?
> > > > > > > > > > > > > > > Will both method `addCatalog` and
> > `registerCatalog`
> > > > be
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > IIUC, in asynchronous registration, it may well
> > > that
> > > > > > CREATE
> > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > executes successfully, but then the following
> > > CREATE
> > > > > > TABLE
> > > > > > > > > > > statement
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > I think it's a break change which should be
> > > > highlighted
> > > > > > in
> > > > > > > > this
> > > > > > > > > > > FLIP,
> > > > > > > > > > > > > may
> > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > BTW, by saying asynchronous registration, I
> would
> > > > like
> > > > > to
> > > > > > > > > expect
> > > > > > > > > > > > there
> > > > > > > > > > > > > > > will be an executor to open or register catalog
> > in
> > > > the
> > > > > > > > > > background,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > "the Catalog instance will be initialized if it
> > has
> > > > not
> > > > > > > been
> > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > yet. If the initialization process fails, these
> > > > > > statements
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > It looks more like lazy initialization for
> > catalog
> > > > than
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence
> > of
> > > > > > Catalog
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi everyone, Thank you all for your interest in
> > > this
> > > > > > > DISCUSS.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > How to handle a catalog with the same name
> that
> > > > > exists
> > > > > > > for
> > > > > > > > > both
> > > > > > > > > > > of
> > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I believe this is a crucial point. Based on my
> > > > current
> > > > > > > > personal
> > > > > > > > > > > > > > > understanding, the Map<String, Catalog>
> catalogs
> > > will
> > > > > > serve
> > > > > > > > as
> > > > > > > > > a
> > > > > > > > > > > > cache
> > > > > > > > > > > > > > > for instantiated catalogs and have the highest
> > > > > priority.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are three methods that can affect the
> > > > Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. registerCatalog(String catalogName, Catalog
> > > > catalog)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This method puts the catalog instance into the
> > > > > > Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This method removes the catalog instance
> > > > corresponding
> > > > > to
> > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This method first retrieves the corresponding
> > > catalog
> > > > > > > > instance
> > > > > > > > > > from
> > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If the
> catalog
> > > > does
> > > > > > not
> > > > > > > > > exist,
> > > > > > > > > > > it
> > > > > > > > > > > > > > > retrieves the corresponding configuration from
> > the
> > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > initializes it, and puts the initialized
> Catalog
> > > > > instance
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The following two methods only modify the
> > > > configuration
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. addCatalog(String catalogName, Map<String,
> > > String>
> > > > > > > > > properties)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This method saves the properties to the
> > > catalogStore
> > > > > and
> > > > > > > > checks
> > > > > > > > > > > > > > > whether there is a catalogName with the same
> > name.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > This method removes the specified configuration
> > of
> > > > the
> > > > > > > > > specified
> > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The following are possible conflict scenarios:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. When the corresponding catalogName already
> > > exists
> > > > in
> > > > > > the
> > > > > > > > > > > > > > > CatalogStore but not in the Map<String,
> Catalog>,
> > > the
> > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > catalog)
> > > > > > method
> > > > > > > > can
> > > > > > > > > > > > > > > succeed and be directly saved to the
> Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. When the corresponding catalogName already
> > > exists
> > > > in
> > > > > > > both
> > > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. When the corresponding catalogName already
> > > exists
> > > > in
> > > > > > the
> > > > > > > > > > > > > > > Map<String, Catalog>, the addCatalog(String
> > > > > catalogName,
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String> properties) method can directly save
> the
> > > > > > properties
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > > catalogName)
> > > > > > method
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > > > use the new properties for initialization
> because
> > > the
> > > > > > > > > > corresponding
> > > > > > > > > > > > > > > catalog instance already exists in catalogs and
> > > will
> > > > be
> > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > Therefore, using the unregisterCatalog(String
> > > > > > catalogName)
> > > > > > > > > method
> > > > > > > > > > > to
> > > > > > > > > > > > > > > remove the instance corresponding to the
> original
> > > > > > > catalogName
> > > > > > > > > is
> > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> `Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs`
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> Map<String,
> > > > > String>
> > > > > > > > > > > properties)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This could potentially lead to confusion. I
> > suggest
> > > > > > > changing
> > > > > > > > > the
> > > > > > > > > > > > > > > method name, perhaps to addCatalog(String
> > > > catalogName,
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String> properties), as previously mentioned
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > add `registerCatalog(String
> catalogName,Catalog
> > > > > > catalog,
> > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Since a catalog is already an instance, adding
> > the
> > > > > > > "lazyInit"
> > > > > > > > > > > > > > > parameter to the registerCatalog(String
> > > catalogName,
> > > > > > > Catalog
> > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > method may not necessarily result in lazy
> > > > > initialization.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think encryption is necessary, but perhaps
> the
> > > > > > encryption
> > > > > > > > > logic
> > > > > > > > > > > > > > > should be implemented in the CatalogStore when
> > the
> > > > data
> > > > > > is
> > > > > > > > > > actually
> > > > > > > > > > > > > > > saved. For instance, the FileCatalogStore could
> > > > encrypt
> > > > > > the
> > > > > > > > > data
> > > > > > > > > > > when
> > > > > > > > > > > > > > > it is saved to a file, while the
> > MemoryCatalogStore
> > > > > does
> > > > > > > not
> > > > > > > > > > > require
> > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it is necessary to have a
> > > MemoryCatalogStore
> > > > as
> > > > > > the
> > > > > > > > > > default
> > > > > > > > > > > > > > > implementation, which saves Catalog
> > configurations
> > > in
> > > > > > > memory.
> > > > > > > > > > > > > > > Otherwise, if we want to implement asynchronous
> > > > loading
> > > > > > of
> > > > > > > > > > Catalog,
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is also a critical concern. When
> executing
> > > DDL,
> > > > > > DQL,
> > > > > > > or
> > > > > > > > > DML
> > > > > > > > > > > > > > > statements that reference a specific Catalog,
> the
> > > > > Catalog
> > > > > > > > > > instance
> > > > > > > > > > > > > > > will be initialized if it has not been
> > initialized
> > > > yet.
> > > > > > If
> > > > > > > > the
> > > > > > > > > > > > > > > initialization process fails, these statements
> > will
> > > > not
> > > > > > be
> > > > > > > > > > executed
> > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs and
> > > > > CatalogStore
> > > > > > > > > > > catalogstore
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the CatalogManager all have the same catalog
> > name,
> > > > but
> > > > > > > > > > correponding
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This issue can be resolved by referring to the
> > > > previous
> > > > > > > > > > responses.
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > key principle is that the Catalog that has been
> > > most
> > > > > > > recently
> > > > > > > > > > used
> > > > > > > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for driving this Flip, I do believe
> this
> > > > Flip
> > > > > > > could
> > > > > > > > be
> > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > for users. The firm I work for also manages a
> lot
> > > of
> > > > > > > > catalogs,
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > submission of tasks becomes slow because of
> > > loading a
> > > > > > > number
> > > > > > > > of
> > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > We obtain the catalogs in the user's Flink SQL
> > > > through
> > > > > > > > regular
> > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > to avoid loading all catalogs to improve the
> > speed
> > > of
> > > > > > task
> > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. When creating a catalog with CREATE
> CATALOG,
> > > the
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > registration method is used by default. What if
> > > > > > > asynchronous
> > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > failed? And how to implement asynchronous
> > > > registration?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. I also have the same question with
> Shammon’s
> > > > first
> > > > > > > > > comment.
> > > > > > > > > > > The
> > > > > > > > > > > > > > > Map<String, Catalog> catalogs and CatalogStore
> > > > > > catalogstore
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > CatalogManager all have the same catalog name,
> > but
> > > > > > > > correponding
> > > > > > > > > > to
> > > > > > > > > > > > > > > different catalog instances. For example:
> > > catalog1->
> > > > > > > > > jdbcCatalog
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the
> > > > > > catalogstore.
> > > > > > > > > Would
> > > > > > > > > > > this
> > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > happen? Maybe we should define `catalogs` and
> > > > > > > `catalogstore`
> > > > > > > > > more
> > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > ruanhang1993@gmail.com
> > > > > >
> > > > > > > 写道：
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, Feng.
I'm trying to understanding the meaning of *lazy initialization*. If i'm wrong, please correct me.
IIUC, lazy initialization means only you need to access the catalog, then you initialize it. But from the code in Proposed Changes, once we register the Catalog,
we initialize it and open it. right?
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Jing Ge" <ji...@ververica.com.INVALID>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Hi Feng,
Thanks for your effort! +1 for the proposal.
One of the major changes is that current design will provide
Map<String,Catalog> catalogs as a snapshot instead of a cache, which means
once it has been initialized, any changes done by other sessions will not
affect it. Point 6 described follow-up options for further improvement.
Best regards,
Jing
On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com> wrote:
> Hi all, I would like to update you on the latest progress of the FLIP.
>
>
> Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang and I
> had an offline discussion regarding the overall solution for Flink
> CatalogStore. We have reached a consensus and I have updated the final
> solution in FLIP.
>
> Next, let me briefly describe the entire design:
>
> 1.
>
> Introduce CatalogDescriptor to store catalog configuration similar to
> TableDescriptor.
> 2.
>
> The two key functions of CatalogStore - void storeCatalog(String
> catalogName, CatalogDescriptor) and CatalogDescriptor getCatalog(String)
> will both use CatalogDescriptor instead of Catalog instance. This way,
> CatalogStore will only be responsible for saving and retrieving catalog
> configurations without having to initialize catalogs.
> 3.
>
> The default registerCatalog(String catalogName, Catalog catalog)
> function in CatalogManager will be marked as deprecated.
> 4.
>
> A new function registerCatalog(String catalogName, CatalogDescriptor
> catalog) will be added to serve as the default registration function for
> catalogs in CatalogManager.
> 5.
>
> Map<String,Catalog> catalogs in CataloManager will remain unchanged and
> save initialized catalogs.This means that deletion operations from one
> session won't synchronize with other sessions.
> 6.
>
> To support multi-session synchronization scenarios for deletions later
> on we should make Map<String,Catalog>catalogs configurable.There may be
> three possible situations:
>
> a.Default caching of all initialized catalogs
>
> b.Introduction of LRU cache logic which can automatically clear
> long-unused catalogs.
>
> c.No caching of any instances; each call to getCatalog creates a new
> instance.
>
>
> This is the document for discussion:
>
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
>
> This is the final proposal document:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>
>
> Thank you very much for your attention and suggestions on this FLIP. A
> special thanks to Hang Ruan for his suggestions on the entire design and
> organizing offline discussions.
>
> If you have any further suggestions or feedback about this FLIP please feel
> free to share.
>
>
> Best,
>
> Feng
>
> On Sat, May 6, 2023 at 8:32 PM Jing Ge <ji...@ververica.com.invalid> wrote:
>
> > Hi Feng,
> >
> > Thanks for improving the FLIP. It looks good to me. We could still
> > reconsider in the future how to provide more common built-in cache
> > functionality in CatalogManager so that not every CatalogSotre
> > implementation has to take care of it.
> >
> > Best regards,
> > Jing
> >
> > On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com> wrote:
> >
> > > Hi Jing,
> > >
> > > Thanks for your reply.
> > >
> > > > There might be more such issues. I would suggest you completely walk
> > > through the FLIP again and fix those issues
> > >
> > > I am very sorry for my carelessness and at the same time, I greatly
> > > appreciate your careful review. I have thoroughly checked the entire
> FLIP
> > > and made corrections to these issues.
> > >
> > >
> > > > If I am not mistaken, with the current FLIP design, CatalogManager
> > > could work without Optional CatalogStore being configured.
> > >
> > > Yes, in the original design, CatalogStore was not necessary because
> > > CatalogManager used Map<String, Catalog> catalogs to store catalog
> > > instances.
> > > However, this caused inconsistency issues. Therefore, I modified this
> > part
> > > of the design and removed Map<String, Catalog> catalogs from
> > > CatalogManager.
> > > At the same time, InMemoryCatalog will serve as the default
> CatalogStore
> > > to save catalogs in memory and replace the functionality of
> > > Map<String,Catalog>catalogs.
> > > The previous plan that kept Map<String,Catalog>catalogs has been moved
> to
> > > Rejected Alternatives.
> > >
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > >
> > > > Hi Feng,
> > > >
> > > > There are still many places contain inconsistent content, e.g.
> > > >
> > > > 1. "Asynchronous registration" is still used.
> > > > 2. The java comment of the method registerCatalog(String catalogName,
> > > > Catalog catalog) in CatalogManager does not tell what the method is
> > > doing.
> > > >
> > > >
> > > > There might be more such issues. I would suggest you completely walk
> > > > through the FLIP again and fix those issues.
> > > >
> > > > About the inMemoryCatalogStore, do you mean that you will build the
> > cache
> > > > functionality in the CatalogStore? This is a very different design
> > > concept
> > > > from what the current FLIP described. If I am not mistaken, with the
> > > > current FLIP design, CatalogManager could work without Optional
> > > > CatalogStore being configured. That is the reason why I mentioned in
> > the
> > > > last email that the example code wrt the Optional CatalogStore is not
> > > > correct.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <ji...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Jing
> > > > >
> > > > >
> > > > > > There are still some NIT issues in the FLIP
> > > > >
> > > > > Thank you very much for the careful review. I have already made the
> > > > > relevant changes.
> > > > >
> > > > >
> > > > > > Speaking of the conflict issues in the multi-instance
> scenarios, I
> > > am
> > > > > not
> > > > > sure if this is the intended behaviour
> > > > >
> > > > > Currently, there are conflicts in multiple scenarios with the
> current
> > > > > design. I am thinking whether we should remove 'Map<String,
> Catalog>'
> > > and
> > > > > make Cache the default behavior of InMemoryCatalogStore. This way,
> > > users
> > > > > can implement their own CatalogStore to achieve multi-instance
> > > > > non-conflicting scenarios. What do you think?
> > > > >
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge <jing@ververica.com.invalid
> >
> > > > > wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > Thanks for working on the FLIP. There are still some NIT issues
> in
> > > the
> > > > > FLIP
> > > > > > like:
> > > > > >
> > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > CatalogStore
> > > > > > instead of Optional in the code example. It should be fine to use
> > it
> > > as
> > > > > > pseudo code for now and update it after you submit the PR.
> > > > > > 2. addCatalog(...) is still used somewhere in the rejected
> section
> > > > which
> > > > > > should be persistContext(...) to keep it consistent.
> > > > > >
> > > > > > Speaking of the conflict issues in the multi-instance scenarios,
> I
> > am
> > > > not
> > > > > > sure if this is the intended behaviour. If Map<String, Catalog>
> > > > catalogs
> > > > > is
> > > > > > used as a cache, it should be invalid, once the related catalog
> has
> > > > been
> > > > > > removed from the CatalogStore by another instance. Did I miss
> > > > something?
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <ji...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Jing,Shammon
> > > > > > > Thanks for your reply.
> > > > > > >
> > > > > > > @Jing
> > > > > > >
> > > > > > > > How about persistCatalog()?
> > > > > > > I think this is a good function name, I have updated it in the
> > > > > > > documentation.
> > > > > > >
> > > > > > > > Some common cache features should be implemented
> > > > > > > Thank you for the suggestion. If alternative 1 turns out to be
> > more
> > > > > > > appropriate later, I will improve this part of the content.
> > > > > > >
> > > > > > > > As the above discussion moves forward, the option 2 solution
> > > looks
> > > > > more
> > > > > > > like a replacement of option 1
> > > > > > > Yes, after discussing with Shammon offline, we think that
> > solution
> > > 2
> > > > > > might
> > > > > > > be more suitable and also avoid any inconsistency issues.
> > > > > > >
> > > > > > > > There are some inconsistent descriptions in the content.
> Would
> > > you
> > > > > > like
> > > > > > > to clean them up?
> > > > > > > I will do my best to improve the document and appreciate your
> > > > > > suggestions.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > @Shammon
> > > > > > > > can you put the unselected option in `Rejected Alternatives`
> > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best
> > > > > > > Feng
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng
> > > > > > > >
> > > > > > > > Thanks for your update.
> > > > > > > >
> > > > > > > > I found there are two options in `Proposed Changes`, can you
> > put
> > > > the
> > > > > > > > unselected option in `Rejected Alternatives`? I think this
> may
> > > help
> > > > > us
> > > > > > > > better understand your proposal
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shammon FY
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > Thanks for raising this FLIP. I am still confused after
> > > > completely
> > > > > > > > reading
> > > > > > > > > the thread with following questions:
> > > > > > > > >
> > > > > > > > > 1. Naming confusion - registerCatalog() and addCatalog()
> have
> > > no
> > > > > big
> > > > > > > > > difference based on their names. One of them is responsible
> > for
> > > > > data
> > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > 2. As you mentioned that Map<String, Catalog> catalogs is
> > used
> > > > as a
> > > > > > > cache
> > > > > > > > > and catalogStore is used for data persistence. I would
> > suggest
> > > > > > > describing
> > > > > > > > > their purpose conceptually and clearly in the FLIP. Some
> > common
> > > > > cache
> > > > > > > > > features should be implemented, i.e. data in the cache and
> > the
> > > > > store
> > > > > > > > should
> > > > > > > > > be consistent. Same Catalog instance should be found in the
> > > store
> > > > > and
> > > > > > > in
> > > > > > > > > the cache(either it has been initialized or it will be lazy
> > > > > > > initialized)
> > > > > > > > > for the same catalog name. The consistency will be taken
> care
> > > of
> > > > > > while
> > > > > > > > > updating the catalog.
> > > > > > > > > 3. As the above discussion moves forward, the option 2
> > solution
> > > > > looks
> > > > > > > > more
> > > > > > > > > like a replacement of option 1, because, afaiu, issues
> > > mentioned
> > > > > > > > > previously with option 1 are not solved yet. Do you still
> > want
> > > to
> > > > > > > propose
> > > > > > > > > both options and ask for suggestions for both of them?
> > > > > > > > > 4. After you updated the FLIP, there are some inconsistent
> > > > > > descriptions
> > > > > > > > in
> > > > > > > > > the content. Would you like to clean them up? Thanks!
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > jinfeng1995@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > hi Shammon
> > > > > > > > > >
> > > > > > > > > > Thank you for your response, and I completely agree with
> > your
> > > > > point
> > > > > > > of
> > > > > > > > > > view.
> > > > > > > > > > Initially, I may have over complicated the whole issue.
> > First
> > > > and
> > > > > > > > > foremost,
> > > > > > > > > > we need to consider the persistence of the Catalog's
> > > > > Configuration.
> > > > > > > > > > If we only need to provide persistence for Catalog
> > > > Configuration,
> > > > > > we
> > > > > > > > can
> > > > > > > > > > add a toConfiguration method to the Catalog interface.
> > > > > > > > > > This method can convert a Catalog instance to a
> Map<String,
> > > > > String>
> > > > > > > > > > properties, and the default implementation will throw an
> > > > > exception.
> > > > > > > > > >
> > > > > > > > > > public interface Catalog {
> > > > > > > > > > /**
> > > > > > > > > > * Returns a map containing the properties of the
> catalog
> > > > > object.
> > > > > > > > > > *
> > > > > > > > > > * @return a map containing the properties of the
> catalog
> > > > > object
> > > > > > > > > > * @throws UnsupportedOperationException if the
> > > implementing
> > > > > > class
> > > > > > > > does
> > > > > > > > > > not override
> > > > > > > > > > * the default implementation of this method
> > > > > > > > > > */
> > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > throw new UnsupportedOperationException("Please
> > implement
> > > > > > > > > toProperties
> > > > > > > > > > for this catalog");
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > The specific process is as follows:
> > > > > > > > > >
> > > > > > > > > > 1. If the user has configured a CatalogStore, the
> > > > > > toConfiguration()
> > > > > > > > > method
> > > > > > > > > > will be called when registering a Catalog instance with
> > > > > > > > > > registerCatalog(String catalogName, Catalog catalog). The
> > > > Catalog
> > > > > > > > > instance
> > > > > > > > > > will be converted to a Map<String, String> properties and
> > > saved
> > > > > to
> > > > > > > the
> > > > > > > > > > CatalogStore. If some Catalog instances do not implement
> > this
> > > > > > method,
> > > > > > > > an
> > > > > > > > > > exception will be thrown.
> > > > > > > > > >
> > > > > > > > > > 2. If the user does not use a CatalogStore, the
> > > > > toConfiguration()
> > > > > > > > method
> > > > > > > > > > will not be called, ensuring consistency with the
> original
> > > > > process.
> > > > > > > > > >
> > > > > > > > > > 3. Saving both the Map<String, Catalog> catalogs and the
> > > > > > CatalogStore
> > > > > > > > at
> > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > For lazy initialization:
> > > > > > > > > > we can start from the Catalog itself and provide a
> > dedicated
> > > > > > Catalog
> > > > > > > > > > implementation for lazy loading, such as
> > ConfigurationCatalog
> > > > > > > > > >
> > > > > > > > > > public class ConfigurationCatalog implements Catalog {
> > > > > > > > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > zjureel@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > >
> > > > > > > > > > > I have no questions about the functionality of
> > > > `CatalogStore`,
> > > > > > but
> > > > > > > > the
> > > > > > > > > > > different operations of multiple `registerCatalog` or
> > > > > > > `storeCatalog`
> > > > > > > > in
> > > > > > > > > > > `CatalogManager`. The implementation in Trino is also
> the
> > > > same,
> > > > > > the
> > > > > > > > > > > `CatalogManager` in trino only has one `createCatalog`,
> > > which
> > > > > > will
> > > > > > > > save
> > > > > > > > > > the
> > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > >
> > > > > > > > > > > I think we don't need to add another `registerCatalog`
> or
> > > > > > > > `addCatalog`
> > > > > > > > > to
> > > > > > > > > > > save a catalog in `Map<String, Catalog> catalogs` or
> > > > > > `CatalogStore
> > > > > > > > > > > catalogStore`. As you mentioned before, the
> `Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How about
> > saving
> > > > the
> > > > > > > > catalog
> > > > > > > > > in
> > > > > > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore
> > > > catalogStore`
> > > > > > > > > together
> > > > > > > > > > > when it is registered or added in `CatalogManager`?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Shammon FY
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > jinfeng1995@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thank you for your reply. I am very sorry for the
> > > > > > > misunderstanding
> > > > > > > > > > caused
> > > > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > > > > > > > > Yes, there was indeed such a discussion before.
> > However,
> > > > in
> > > > > > > > > designing
> > > > > > > > > > > the
> > > > > > > > > > > > whole solution, I found that the logic of
> > CatalogManager
> > > > > itself
> > > > > > > > > doesn't
> > > > > > > > > > > > need to change much. *We cannot only persist Catalog
> > > > > instances
> > > > > > > > > > > themselves*,
> > > > > > > > > > > > so exposing only registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > might
> > > > > > > > > > > > not be enough to save Catalogs, because in the end we
> > > still
> > > > > > need
> > > > > > > to
> > > > > > > > > > save
> > > > > > > > > > > > the configurations corresponding to the Catalog
> > > instances.
> > > > > > > > > Therefore,
> > > > > > > > > > I
> > > > > > > > > > > > decided to introduce the CatalogStore interface for
> > > > > > configuration
> > > > > > > > > > > > persistence. Regarding this part, I also took
> > inspiration
> > > > > from
> > > > > > > > > Trino's
> > > > > > > > > > > > implementation[1].
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > @yuxia
> > > > > > > > > > > >
> > > > > > > > > > > > > 1: The mechanism of handling catalog with the same
> > name
> > > > > > looks a
> > > > > > > > > > little
> > > > > > > > > > > of
> > > > > > > > > > > > complex to me.
> > > > > > > > > > > > Thank you for the suggestion. I will provide a
> detailed
> > > > > > > description
> > > > > > > > > and
> > > > > > > > > > > > code examples for this part, and add it to the FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> > > confused
> > > > > to
> > > > > > > me.
> > > > > > > > > > IIUC,
> > > > > > > > > > > > this method is for storing catalog to CatalogStore,
> how
> > > > about
> > > > > > > > > renaming
> > > > > > > > > > it
> > > > > > > > > > > > to `storeCatalog`? It's very personal opinion, you
> can
> > > > decide
> > > > > > to
> > > > > > > > take
> > > > > > > > > > it
> > > > > > > > > > > or
> > > > > > > > > > > > not by your self.
> > > > > > > > > > > > StoreCatalog looks more intuitive to me. I don't see
> > any
> > > > > > problem
> > > > > > > > with
> > > > > > > > > > it.
> > > > > > > > > > > >
> > > > > > > > > > > > > 3.For CREATE CATALOG statement, which method will
> be
> > > > > called?
> > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> > whether
> > > > user
> > > > > > can
> > > > > > > > > add a
> > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > For CREATE CATALOG, my original design was to add it
> > > > directly
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > CatalogStore, but this would disrupt the existing
> > logic.
> > > > > > > > Therefore, I
> > > > > > > > > > > think
> > > > > > > > > > > > we can do both: save the configuration to the
> > > CatalogStore
> > > > > and
> > > > > > > > > > initialize
> > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > >
> > > > > > > > > > > > > 3. Is it really neccessary to provide a default
> > > > > implmentation
> > > > > > > for
> > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > I think it is necessary, otherwise we would need to
> > > > introduce
> > > > > > an
> > > > > > > > > > > additional
> > > > > > > > > > > > Map to store the configuration for lazy loading.
> > > > > > > > > > > >
> > > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > > I don't think registerCatalog(String catalogName,
> > Catalog
> > > > > > > catalog)
> > > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > made into an asynchronous interface because Catalog
> is
> > > > > already
> > > > > > an
> > > > > > > > > > > instance.
> > > > > > > > > > > >
> > > > > > > > > > > > > It looks more like lazy initialization for catalog
> > than
> > > > > > > > > asynchronous
> > > > > > > > > > > > registration, right?
> > > > > > > > > > > > Indeed, my description was inaccurate. It should be
> > lazy
> > > > > > > > registration
> > > > > > > > > > > > instead of asynchronous registration. I have already
> > > > updated
> > > > > > the
> > > > > > > > > title
> > > > > > > > > > of
> > > > > > > > > > > > the FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > [1].
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > zjureel@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think if there is a `registerCatalog` method in
> > > > > > > > `CatalogManager`,
> > > > > > > > > > it
> > > > > > > > > > > > will
> > > > > > > > > > > > > confuse users whether a method named `addCatalog`
> or
> > > > > > > > `storeCatalog`
> > > > > > > > > > is
> > > > > > > > > > > > > added.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And as you mentioned, the memory catalog is a
> > `cache`,
> > > I
> > > > > > think
> > > > > > > > the
> > > > > > > > > > > > concept
> > > > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I found there is a pre-discussion [1] for this
> FLIP.
> > > > Please
> > > > > > > > correct
> > > > > > > > > > me
> > > > > > > > > > > if
> > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that discussion
> is
> > > to
> > > > > use
> > > > > > > > > > > > > `CatalogManager` as an interface and implement it
> for
> > > > > > different
> > > > > > > > > > stores
> > > > > > > > > > > > such
> > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think there is a gap between the current FLIP
> > design
> > > > and
> > > > > > that
> > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > What about the proposal of the discussion in thread
> > > [1] ?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > > > > > > > >
> > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > 1: The mechanism of handling catalog with the
> same
> > > name
> > > > > > > looks a
> > > > > > > > > > > little
> > > > > > > > > > > > of
> > > > > > > > > > > > > > complex to me. I think it'll be better to explain
> > it
> > > in
> > > > > the
> > > > > > > > java
> > > > > > > > > > doc
> > > > > > > > > > > of
> > > > > > > > > > > > > > these methods and give a brief example in this
> > FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> > > > confused
> > > > > > to
> > > > > > > > me.
> > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > this method is for storing catalog to
> CatalogStore,
> > > how
> > > > > > about
> > > > > > > > > > > renaming
> > > > > > > > > > > > it
> > > > > > > > > > > > > > to `storeCatalog`? It's very personal opinion,
> you
> > > can
> > > > > > decide
> > > > > > > > to
> > > > > > > > > > take
> > > > > > > > > > > > it
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which method
> will
> > be
> > > > > > called?
> > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> > > > whether
> > > > > > user
> > > > > > > > can
> > > > > > > > > > > add a
> > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3: Is it really neccessary to provide a default
> > > > > > implmentation
> > > > > > > > for
> > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > When creating a catalog with CREATE CATALOG,
> the
> > > > > > > asynchronous
> > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > If asynchronous registration is the default
> > behavior,
> > > > it
> > > > > > > there
> > > > > > > > > any
> > > > > > > > > > > way
> > > > > > > > > > > > > > that user can switch to synchronous registration
> > just
> > > > > like
> > > > > > > > > before?
> > > > > > > > > > > > > > Will both method `addCatalog` and
> `registerCatalog`
> > > be
> > > > > > > > > asynchronous
> > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > IIUC, in asynchronous registration, it may well
> > that
> > > > > CREATE
> > > > > > > > > CATALOG
> > > > > > > > > > > > > > executes successfully, but then the following
> > CREATE
> > > > > TABLE
> > > > > > > > > > statement
> > > > > > > > > > > > will
> > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > I think it's a break change which should be
> > > highlighted
> > > > > in
> > > > > > > this
> > > > > > > > > > FLIP,
> > > > > > > > > > > > may
> > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > BTW, by saying asynchronous registration, I would
> > > like
> > > > to
> > > > > > > > expect
> > > > > > > > > > > there
> > > > > > > > > > > > > > will be an executor to open or register catalog
> in
> > > the
> > > > > > > > > background,
> > > > > > > > > > > but
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > "the Catalog instance will be initialized if it
> has
> > > not
> > > > > > been
> > > > > > > > > > > > initialized
> > > > > > > > > > > > > > yet. If the initialization process fails, these
> > > > > statements
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > be
> > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > It looks more like lazy initialization for
> catalog
> > > than
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence
> of
> > > > > Catalog
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone, Thank you all for your interest in
> > this
> > > > > > DISCUSS.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > How to handle a catalog with the same name that
> > > > exists
> > > > > > for
> > > > > > > > both
> > > > > > > > > > of
> > > > > > > > > > > > > them?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I believe this is a crucial point. Based on my
> > > current
> > > > > > > personal
> > > > > > > > > > > > > > understanding, the Map<String, Catalog> catalogs
> > will
> > > > > serve
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > > cache
> > > > > > > > > > > > > > for instantiated catalogs and have the highest
> > > > priority.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There are three methods that can affect the
> > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. registerCatalog(String catalogName, Catalog
> > > catalog)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This method puts the catalog instance into the
> > > > > Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This method removes the catalog instance
> > > corresponding
> > > > to
> > > > > > > > > > catalogName
> > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This method first retrieves the corresponding
> > catalog
> > > > > > > instance
> > > > > > > > > from
> > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If the catalog
> > > does
> > > > > not
> > > > > > > > exist,
> > > > > > > > > > it
> > > > > > > > > > > > > > retrieves the corresponding configuration from
> the
> > > > > > > > CatalogStore,
> > > > > > > > > > > > > > initializes it, and puts the initialized Catalog
> > > > instance
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The following two methods only modify the
> > > configuration
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. addCatalog(String catalogName, Map<String,
> > String>
> > > > > > > > properties)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This method saves the properties to the
> > catalogStore
> > > > and
> > > > > > > checks
> > > > > > > > > > > > > > whether there is a catalogName with the same
> name.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > This method removes the specified configuration
> of
> > > the
> > > > > > > > specified
> > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The following are possible conflict scenarios:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. When the corresponding catalogName already
> > exists
> > > in
> > > > > the
> > > > > > > > > > > > > > CatalogStore but not in the Map<String, Catalog>,
> > the
> > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > catalog)
> > > > > method
> > > > > > > can
> > > > > > > > > > > > > > succeed and be directly saved to the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. When the corresponding catalogName already
> > exists
> > > in
> > > > > > both
> > > > > > > > the
> > > > > > > > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. When the corresponding catalogName already
> > exists
> > > in
> > > > > the
> > > > > > > > > > > > > > Map<String, Catalog>, the addCatalog(String
> > > > catalogName,
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > String> properties) method can directly save the
> > > > > properties
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > catalogName)
> > > > > method
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > > > use the new properties for initialization because
> > the
> > > > > > > > > corresponding
> > > > > > > > > > > > > > catalog instance already exists in catalogs and
> > will
> > > be
> > > > > > > > > > prioritized.
> > > > > > > > > > > > > > Therefore, using the unregisterCatalog(String
> > > > > catalogName)
> > > > > > > > method
> > > > > > > > > > to
> > > > > > > > > > > > > > remove the instance corresponding to the original
> > > > > > catalogName
> > > > > > > > is
> > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > `registerCatalog(String
> > > > > > > > > > > > > > catalogName,Catalog catalog)` in the `Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs`
> > > > > > > > > > > > and
> > > > > > > > > > > > > > `registerCatalog(String catalogName, Map<String,
> > > > String>
> > > > > > > > > > properties)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This could potentially lead to confusion. I
> suggest
> > > > > > changing
> > > > > > > > the
> > > > > > > > > > > > > > method name, perhaps to addCatalog(String
> > > catalogName,
> > > > > > > > > Map<String,
> > > > > > > > > > > > > > String> properties), as previously mentioned
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > add `registerCatalog(String catalogName,Catalog
> > > > > catalog,
> > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Since a catalog is already an instance, adding
> the
> > > > > > "lazyInit"
> > > > > > > > > > > > > > parameter to the registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > catalog)
> > > > > > > > > > > > > > method may not necessarily result in lazy
> > > > initialization.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think encryption is necessary, but perhaps the
> > > > > encryption
> > > > > > > > logic
> > > > > > > > > > > > > > should be implemented in the CatalogStore when
> the
> > > data
> > > > > is
> > > > > > > > > actually
> > > > > > > > > > > > > > saved. For instance, the FileCatalogStore could
> > > encrypt
> > > > > the
> > > > > > > > data
> > > > > > > > > > when
> > > > > > > > > > > > > > it is saved to a file, while the
> MemoryCatalogStore
> > > > does
> > > > > > not
> > > > > > > > > > require
> > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it is necessary to have a
> > MemoryCatalogStore
> > > as
> > > > > the
> > > > > > > > > default
> > > > > > > > > > > > > > implementation, which saves Catalog
> configurations
> > in
> > > > > > memory.
> > > > > > > > > > > > > > Otherwise, if we want to implement asynchronous
> > > loading
> > > > > of
> > > > > > > > > Catalog,
> > > > > > > > > > > we
> > > > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is also a critical concern. When executing
> > DDL,
> > > > > DQL,
> > > > > > or
> > > > > > > > DML
> > > > > > > > > > > > > > statements that reference a specific Catalog, the
> > > > Catalog
> > > > > > > > > instance
> > > > > > > > > > > > > > will be initialized if it has not been
> initialized
> > > yet.
> > > > > If
> > > > > > > the
> > > > > > > > > > > > > > initialization process fails, these statements
> will
> > > not
> > > > > be
> > > > > > > > > executed
> > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs and
> > > > CatalogStore
> > > > > > > > > > catalogstore
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the CatalogManager all have the same catalog
> name,
> > > but
> > > > > > > > > correponding
> > > > > > > > > > > to
> > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This issue can be resolved by referring to the
> > > previous
> > > > > > > > > responses.
> > > > > > > > > > > The
> > > > > > > > > > > > > > key principle is that the Catalog that has been
> > most
> > > > > > recently
> > > > > > > > > used
> > > > > > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for driving this Flip, I do believe this
> > > Flip
> > > > > > could
> > > > > > > be
> > > > > > > > > > > helpful
> > > > > > > > > > > > > > for users. The firm I work for also manages a lot
> > of
> > > > > > > catalogs,
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > > submission of tasks becomes slow because of
> > loading a
> > > > > > number
> > > > > > > of
> > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > We obtain the catalogs in the user's Flink SQL
> > > through
> > > > > > > regular
> > > > > > > > > > > > expression
> > > > > > > > > > > > > > to avoid loading all catalogs to improve the
> speed
> > of
> > > > > task
> > > > > > > > > > > submission.
> > > > > > > > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. When creating a catalog with CREATE CATALOG,
> > the
> > > > > > > > > asynchronous
> > > > > > > > > > > > > > registration method is used by default. What if
> > > > > > asynchronous
> > > > > > > > > > > > registration
> > > > > > > > > > > > > > failed? And how to implement asynchronous
> > > registration?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. I also have the same question with Shammon’s
> > > first
> > > > > > > > comment.
> > > > > > > > > > The
> > > > > > > > > > > > > > Map<String, Catalog> catalogs and CatalogStore
> > > > > catalogstore
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > CatalogManager all have the same catalog name,
> but
> > > > > > > correponding
> > > > > > > > > to
> > > > > > > > > > > > > > different catalog instances. For example:
> > catalog1->
> > > > > > > > jdbcCatalog
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the
> > > > > catalogstore.
> > > > > > > > Would
> > > > > > > > > > this
> > > > > > > > > > > > > case
> > > > > > > > > > > > > > happen? Maybe we should define `catalogs` and
> > > > > > `catalogstore`
> > > > > > > > more
> > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > ruanhang1993@gmail.com
> > > > >
> > > > > > 写道：
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Feng,
Thanks for your effort! +1 for the proposal.
One of the major changes is that current design will provide
Map<String,Catalog> catalogs as a snapshot instead of a cache, which means
once it has been initialized, any changes done by other sessions will not
affect it. Point 6 described follow-up options for further improvement.
Best regards,
Jing
On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com> wrote:
> Hi all, I would like to update you on the latest progress of the FLIP.
>
>
> Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang and I
> had an offline discussion regarding the overall solution for Flink
> CatalogStore. We have reached a consensus and I have updated the final
> solution in FLIP.
>
> Next, let me briefly describe the entire design:
>
> 1.
>
> Introduce CatalogDescriptor to store catalog configuration similar to
> TableDescriptor.
> 2.
>
> The two key functions of CatalogStore - void storeCatalog(String
> catalogName, CatalogDescriptor) and CatalogDescriptor getCatalog(String)
> will both use CatalogDescriptor instead of Catalog instance. This way,
> CatalogStore will only be responsible for saving and retrieving catalog
> configurations without having to initialize catalogs.
> 3.
>
> The default registerCatalog(String catalogName, Catalog catalog)
> function in CatalogManager will be marked as deprecated.
> 4.
>
> A new function registerCatalog(String catalogName, CatalogDescriptor
> catalog) will be added to serve as the default registration function for
> catalogs in CatalogManager.
> 5.
>
> Map<String,Catalog> catalogs in CataloManager will remain unchanged and
> save initialized catalogs.This means that deletion operations from one
> session won't synchronize with other sessions.
> 6.
>
> To support multi-session synchronization scenarios for deletions later
> on we should make Map<String,Catalog>catalogs configurable.There may be
> three possible situations:
>
> a.Default caching of all initialized catalogs
>
> b.Introduction of LRU cache logic which can automatically clear
> long-unused catalogs.
>
> c.No caching of any instances; each call to getCatalog creates a new
> instance.
>
>
> This is the document for discussion:
>
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
>
> This is the final proposal document:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>
>
> Thank you very much for your attention and suggestions on this FLIP. A
> special thanks to Hang Ruan for his suggestions on the entire design and
> organizing offline discussions.
>
> If you have any further suggestions or feedback about this FLIP please feel
> free to share.
>
>
> Best,
>
> Feng
>
> On Sat, May 6, 2023 at 8:32 PM Jing Ge <ji...@ververica.com.invalid> wrote:
>
> > Hi Feng,
> >
> > Thanks for improving the FLIP. It looks good to me. We could still
> > reconsider in the future how to provide more common built-in cache
> > functionality in CatalogManager so that not every CatalogSotre
> > implementation has to take care of it.
> >
> > Best regards,
> > Jing
> >
> > On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com> wrote:
> >
> > > Hi Jing,
> > >
> > > Thanks for your reply.
> > >
> > > > There might be more such issues. I would suggest you completely walk
> > > through the FLIP again and fix those issues
> > >
> > > I am very sorry for my carelessness and at the same time, I greatly
> > > appreciate your careful review. I have thoroughly checked the entire
> FLIP
> > > and made corrections to these issues.
> > >
> > >
> > > > If I am not mistaken, with the current FLIP design, CatalogManager
> > > could work without Optional CatalogStore being configured.
> > >
> > > Yes, in the original design, CatalogStore was not necessary because
> > > CatalogManager used Map<String, Catalog> catalogs to store catalog
> > > instances.
> > > However, this caused inconsistency issues. Therefore, I modified this
> > part
> > > of the design and removed Map<String, Catalog> catalogs from
> > > CatalogManager.
> > > At the same time, InMemoryCatalog will serve as the default
> CatalogStore
> > > to save catalogs in memory and replace the functionality of
> > > Map<String,Catalog>catalogs.
> > > The previous plan that kept Map<String,Catalog>catalogs has been moved
> to
> > > Rejected Alternatives.
> > >
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > >
> > > > Hi Feng,
> > > >
> > > > There are still many places contain inconsistent content, e.g.
> > > >
> > > > 1. "Asynchronous registration" is still used.
> > > > 2. The java comment of the method registerCatalog(String catalogName,
> > > > Catalog catalog) in CatalogManager does not tell what the method is
> > > doing.
> > > >
> > > >
> > > > There might be more such issues. I would suggest you completely walk
> > > > through the FLIP again and fix those issues.
> > > >
> > > > About the inMemoryCatalogStore, do you mean that you will build the
> > cache
> > > > functionality in the CatalogStore? This is a very different design
> > > concept
> > > > from what the current FLIP described. If I am not mistaken, with the
> > > > current FLIP design, CatalogManager could work without Optional
> > > > CatalogStore being configured. That is the reason why I mentioned in
> > the
> > > > last email that the example code wrt the Optional CatalogStore is not
> > > > correct.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <ji...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Jing
> > > > >
> > > > >
> > > > > > There are still some NIT issues in the FLIP
> > > > >
> > > > > Thank you very much for the careful review. I have already made the
> > > > > relevant changes.
> > > > >
> > > > >
> > > > > > Speaking of the conflict issues in the multi-instance
> scenarios, I
> > > am
> > > > > not
> > > > > sure if this is the intended behaviour
> > > > >
> > > > > Currently, there are conflicts in multiple scenarios with the
> current
> > > > > design. I am thinking whether we should remove 'Map<String,
> Catalog>'
> > > and
> > > > > make Cache the default behavior of InMemoryCatalogStore. This way,
> > > users
> > > > > can implement their own CatalogStore to achieve multi-instance
> > > > > non-conflicting scenarios. What do you think?
> > > > >
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge <jing@ververica.com.invalid
> >
> > > > > wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > Thanks for working on the FLIP. There are still some NIT issues
> in
> > > the
> > > > > FLIP
> > > > > > like:
> > > > > >
> > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > CatalogStore
> > > > > > instead of Optional in the code example. It should be fine to use
> > it
> > > as
> > > > > > pseudo code for now and update it after you submit the PR.
> > > > > > 2. addCatalog(...) is still used somewhere in the rejected
> section
> > > > which
> > > > > > should be persistContext(...) to keep it consistent.
> > > > > >
> > > > > > Speaking of the conflict issues in the multi-instance scenarios,
> I
> > am
> > > > not
> > > > > > sure if this is the intended behaviour. If Map<String, Catalog>
> > > > catalogs
> > > > > is
> > > > > > used as a cache, it should be invalid, once the related catalog
> has
> > > > been
> > > > > > removed from the CatalogStore by another instance. Did I miss
> > > > something?
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <ji...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Jing,Shammon
> > > > > > > Thanks for your reply.
> > > > > > >
> > > > > > > @Jing
> > > > > > >
> > > > > > > > How about persistCatalog()?
> > > > > > > I think this is a good function name, I have updated it in the
> > > > > > > documentation.
> > > > > > >
> > > > > > > > Some common cache features should be implemented
> > > > > > > Thank you for the suggestion. If alternative 1 turns out to be
> > more
> > > > > > > appropriate later, I will improve this part of the content.
> > > > > > >
> > > > > > > > As the above discussion moves forward, the option 2 solution
> > > looks
> > > > > more
> > > > > > > like a replacement of option 1
> > > > > > > Yes, after discussing with Shammon offline, we think that
> > solution
> > > 2
> > > > > > might
> > > > > > > be more suitable and also avoid any inconsistency issues.
> > > > > > >
> > > > > > > > There are some inconsistent descriptions in the content.
> Would
> > > you
> > > > > > like
> > > > > > > to clean them up?
> > > > > > > I will do my best to improve the document and appreciate your
> > > > > > suggestions.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > @Shammon
> > > > > > > > can you put the unselected option in `Rejected Alternatives`
> > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best
> > > > > > > Feng
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng
> > > > > > > >
> > > > > > > > Thanks for your update.
> > > > > > > >
> > > > > > > > I found there are two options in `Proposed Changes`, can you
> > put
> > > > the
> > > > > > > > unselected option in `Rejected Alternatives`? I think this
> may
> > > help
> > > > > us
> > > > > > > > better understand your proposal
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shammon FY
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > Thanks for raising this FLIP. I am still confused after
> > > > completely
> > > > > > > > reading
> > > > > > > > > the thread with following questions:
> > > > > > > > >
> > > > > > > > > 1. Naming confusion - registerCatalog() and addCatalog()
> have
> > > no
> > > > > big
> > > > > > > > > difference based on their names. One of them is responsible
> > for
> > > > > data
> > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > 2. As you mentioned that Map<String, Catalog> catalogs is
> > used
> > > > as a
> > > > > > > cache
> > > > > > > > > and catalogStore is used for data persistence. I would
> > suggest
> > > > > > > describing
> > > > > > > > > their purpose conceptually and clearly in the FLIP. Some
> > common
> > > > > cache
> > > > > > > > > features should be implemented, i.e. data in the cache and
> > the
> > > > > store
> > > > > > > > should
> > > > > > > > > be consistent. Same Catalog instance should be found in the
> > > store
> > > > > and
> > > > > > > in
> > > > > > > > > the cache(either it has been initialized or it will be lazy
> > > > > > > initialized)
> > > > > > > > > for the same catalog name. The consistency will be taken
> care
> > > of
> > > > > > while
> > > > > > > > > updating the catalog.
> > > > > > > > > 3. As the above discussion moves forward, the option 2
> > solution
> > > > > looks
> > > > > > > > more
> > > > > > > > > like a replacement of option 1, because, afaiu, issues
> > > mentioned
> > > > > > > > > previously with option 1 are not solved yet. Do you still
> > want
> > > to
> > > > > > > propose
> > > > > > > > > both options and ask for suggestions for both of them?
> > > > > > > > > 4. After you updated the FLIP, there are some inconsistent
> > > > > > descriptions
> > > > > > > > in
> > > > > > > > > the content. Would you like to clean them up? Thanks!
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > jinfeng1995@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > hi Shammon
> > > > > > > > > >
> > > > > > > > > > Thank you for your response, and I completely agree with
> > your
> > > > > point
> > > > > > > of
> > > > > > > > > > view.
> > > > > > > > > > Initially, I may have over complicated the whole issue.
> > First
> > > > and
> > > > > > > > > foremost,
> > > > > > > > > > we need to consider the persistence of the Catalog's
> > > > > Configuration.
> > > > > > > > > > If we only need to provide persistence for Catalog
> > > > Configuration,
> > > > > > we
> > > > > > > > can
> > > > > > > > > > add a toConfiguration method to the Catalog interface.
> > > > > > > > > > This method can convert a Catalog instance to a
> Map<String,
> > > > > String>
> > > > > > > > > > properties, and the default implementation will throw an
> > > > > exception.
> > > > > > > > > >
> > > > > > > > > > public interface Catalog {
> > > > > > > > > > /**
> > > > > > > > > > * Returns a map containing the properties of the
> catalog
> > > > > object.
> > > > > > > > > > *
> > > > > > > > > > * @return a map containing the properties of the
> catalog
> > > > > object
> > > > > > > > > > * @throws UnsupportedOperationException if the
> > > implementing
> > > > > > class
> > > > > > > > does
> > > > > > > > > > not override
> > > > > > > > > > * the default implementation of this method
> > > > > > > > > > */
> > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > throw new UnsupportedOperationException("Please
> > implement
> > > > > > > > > toProperties
> > > > > > > > > > for this catalog");
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > The specific process is as follows:
> > > > > > > > > >
> > > > > > > > > > 1. If the user has configured a CatalogStore, the
> > > > > > toConfiguration()
> > > > > > > > > method
> > > > > > > > > > will be called when registering a Catalog instance with
> > > > > > > > > > registerCatalog(String catalogName, Catalog catalog). The
> > > > Catalog
> > > > > > > > > instance
> > > > > > > > > > will be converted to a Map<String, String> properties and
> > > saved
> > > > > to
> > > > > > > the
> > > > > > > > > > CatalogStore. If some Catalog instances do not implement
> > this
> > > > > > method,
> > > > > > > > an
> > > > > > > > > > exception will be thrown.
> > > > > > > > > >
> > > > > > > > > > 2. If the user does not use a CatalogStore, the
> > > > > toConfiguration()
> > > > > > > > method
> > > > > > > > > > will not be called, ensuring consistency with the
> original
> > > > > process.
> > > > > > > > > >
> > > > > > > > > > 3. Saving both the Map<String, Catalog> catalogs and the
> > > > > > CatalogStore
> > > > > > > > at
> > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > For lazy initialization:
> > > > > > > > > > we can start from the Catalog itself and provide a
> > dedicated
> > > > > > Catalog
> > > > > > > > > > implementation for lazy loading, such as
> > ConfigurationCatalog
> > > > > > > > > >
> > > > > > > > > > public class ConfigurationCatalog implements Catalog {
> > > > > > > > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > zjureel@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > >
> > > > > > > > > > > I have no questions about the functionality of
> > > > `CatalogStore`,
> > > > > > but
> > > > > > > > the
> > > > > > > > > > > different operations of multiple `registerCatalog` or
> > > > > > > `storeCatalog`
> > > > > > > > in
> > > > > > > > > > > `CatalogManager`. The implementation in Trino is also
> the
> > > > same,
> > > > > > the
> > > > > > > > > > > `CatalogManager` in trino only has one `createCatalog`,
> > > which
> > > > > > will
> > > > > > > > save
> > > > > > > > > > the
> > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > >
> > > > > > > > > > > I think we don't need to add another `registerCatalog`
> or
> > > > > > > > `addCatalog`
> > > > > > > > > to
> > > > > > > > > > > save a catalog in `Map<String, Catalog> catalogs` or
> > > > > > `CatalogStore
> > > > > > > > > > > catalogStore`. As you mentioned before, the
> `Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How about
> > saving
> > > > the
> > > > > > > > catalog
> > > > > > > > > in
> > > > > > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore
> > > > catalogStore`
> > > > > > > > > together
> > > > > > > > > > > when it is registered or added in `CatalogManager`?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Shammon FY
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > jinfeng1995@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thank you for your reply. I am very sorry for the
> > > > > > > misunderstanding
> > > > > > > > > > caused
> > > > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > > > >
> > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > > > > > > > > Yes, there was indeed such a discussion before.
> > However,
> > > > in
> > > > > > > > > designing
> > > > > > > > > > > the
> > > > > > > > > > > > whole solution, I found that the logic of
> > CatalogManager
> > > > > itself
> > > > > > > > > doesn't
> > > > > > > > > > > > need to change much. *We cannot only persist Catalog
> > > > > instances
> > > > > > > > > > > themselves*,
> > > > > > > > > > > > so exposing only registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > might
> > > > > > > > > > > > not be enough to save Catalogs, because in the end we
> > > still
> > > > > > need
> > > > > > > to
> > > > > > > > > > save
> > > > > > > > > > > > the configurations corresponding to the Catalog
> > > instances.
> > > > > > > > > Therefore,
> > > > > > > > > > I
> > > > > > > > > > > > decided to introduce the CatalogStore interface for
> > > > > > configuration
> > > > > > > > > > > > persistence. Regarding this part, I also took
> > inspiration
> > > > > from
> > > > > > > > > Trino's
> > > > > > > > > > > > implementation[1].
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > @yuxia
> > > > > > > > > > > >
> > > > > > > > > > > > > 1: The mechanism of handling catalog with the same
> > name
> > > > > > looks a
> > > > > > > > > > little
> > > > > > > > > > > of
> > > > > > > > > > > > complex to me.
> > > > > > > > > > > > Thank you for the suggestion. I will provide a
> detailed
> > > > > > > description
> > > > > > > > > and
> > > > > > > > > > > > code examples for this part, and add it to the FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> > > confused
> > > > > to
> > > > > > > me.
> > > > > > > > > > IIUC,
> > > > > > > > > > > > this method is for storing catalog to CatalogStore,
> how
> > > > about
> > > > > > > > > renaming
> > > > > > > > > > it
> > > > > > > > > > > > to `storeCatalog`? It's very personal opinion, you
> can
> > > > decide
> > > > > > to
> > > > > > > > take
> > > > > > > > > > it
> > > > > > > > > > > or
> > > > > > > > > > > > not by your self.
> > > > > > > > > > > > StoreCatalog looks more intuitive to me. I don't see
> > any
> > > > > > problem
> > > > > > > > with
> > > > > > > > > > it.
> > > > > > > > > > > >
> > > > > > > > > > > > > 3.For CREATE CATALOG statement, which method will
> be
> > > > > called?
> > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> > whether
> > > > user
> > > > > > can
> > > > > > > > > add a
> > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > For CREATE CATALOG, my original design was to add it
> > > > directly
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > CatalogStore, but this would disrupt the existing
> > logic.
> > > > > > > > Therefore, I
> > > > > > > > > > > think
> > > > > > > > > > > > we can do both: save the configuration to the
> > > CatalogStore
> > > > > and
> > > > > > > > > > initialize
> > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > >
> > > > > > > > > > > > > 3. Is it really neccessary to provide a default
> > > > > implmentation
> > > > > > > for
> > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > I think it is necessary, otherwise we would need to
> > > > introduce
> > > > > > an
> > > > > > > > > > > additional
> > > > > > > > > > > > Map to store the configuration for lazy loading.
> > > > > > > > > > > >
> > > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > > I don't think registerCatalog(String catalogName,
> > Catalog
> > > > > > > catalog)
> > > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > made into an asynchronous interface because Catalog
> is
> > > > > already
> > > > > > an
> > > > > > > > > > > instance.
> > > > > > > > > > > >
> > > > > > > > > > > > > It looks more like lazy initialization for catalog
> > than
> > > > > > > > > asynchronous
> > > > > > > > > > > > registration, right?
> > > > > > > > > > > > Indeed, my description was inaccurate. It should be
> > lazy
> > > > > > > > registration
> > > > > > > > > > > > instead of asynchronous registration. I have already
> > > > updated
> > > > > > the
> > > > > > > > > title
> > > > > > > > > > of
> > > > > > > > > > > > the FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > [1].
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > zjureel@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think if there is a `registerCatalog` method in
> > > > > > > > `CatalogManager`,
> > > > > > > > > > it
> > > > > > > > > > > > will
> > > > > > > > > > > > > confuse users whether a method named `addCatalog`
> or
> > > > > > > > `storeCatalog`
> > > > > > > > > > is
> > > > > > > > > > > > > added.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And as you mentioned, the memory catalog is a
> > `cache`,
> > > I
> > > > > > think
> > > > > > > > the
> > > > > > > > > > > > concept
> > > > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I found there is a pre-discussion [1] for this
> FLIP.
> > > > Please
> > > > > > > > correct
> > > > > > > > > > me
> > > > > > > > > > > if
> > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that discussion
> is
> > > to
> > > > > use
> > > > > > > > > > > > > `CatalogManager` as an interface and implement it
> for
> > > > > > different
> > > > > > > > > > stores
> > > > > > > > > > > > such
> > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think there is a gap between the current FLIP
> > design
> > > > and
> > > > > > that
> > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > What about the proposal of the discussion in thread
> > > [1] ?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > > > > > > > >
> > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > 1: The mechanism of handling catalog with the
> same
> > > name
> > > > > > > looks a
> > > > > > > > > > > little
> > > > > > > > > > > > of
> > > > > > > > > > > > > > complex to me. I think it'll be better to explain
> > it
> > > in
> > > > > the
> > > > > > > > java
> > > > > > > > > > doc
> > > > > > > > > > > of
> > > > > > > > > > > > > > these methods and give a brief example in this
> > FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> > > > confused
> > > > > > to
> > > > > > > > me.
> > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > this method is for storing catalog to
> CatalogStore,
> > > how
> > > > > > about
> > > > > > > > > > > renaming
> > > > > > > > > > > > it
> > > > > > > > > > > > > > to `storeCatalog`? It's very personal opinion,
> you
> > > can
> > > > > > decide
> > > > > > > > to
> > > > > > > > > > take
> > > > > > > > > > > > it
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which method
> will
> > be
> > > > > > called?
> > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> > > > whether
> > > > > > user
> > > > > > > > can
> > > > > > > > > > > add a
> > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3: Is it really neccessary to provide a default
> > > > > > implmentation
> > > > > > > > for
> > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > When creating a catalog with CREATE CATALOG,
> the
> > > > > > > asynchronous
> > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > If asynchronous registration is the default
> > behavior,
> > > > it
> > > > > > > there
> > > > > > > > > any
> > > > > > > > > > > way
> > > > > > > > > > > > > > that user can switch to synchronous registration
> > just
> > > > > like
> > > > > > > > > before?
> > > > > > > > > > > > > > Will both method `addCatalog` and
> `registerCatalog`
> > > be
> > > > > > > > > asynchronous
> > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > IIUC, in asynchronous registration, it may well
> > that
> > > > > CREATE
> > > > > > > > > CATALOG
> > > > > > > > > > > > > > executes successfully, but then the following
> > CREATE
> > > > > TABLE
> > > > > > > > > > statement
> > > > > > > > > > > > will
> > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > I think it's a break change which should be
> > > highlighted
> > > > > in
> > > > > > > this
> > > > > > > > > > FLIP,
> > > > > > > > > > > > may
> > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > BTW, by saying asynchronous registration, I would
> > > like
> > > > to
> > > > > > > > expect
> > > > > > > > > > > there
> > > > > > > > > > > > > > will be an executor to open or register catalog
> in
> > > the
> > > > > > > > > background,
> > > > > > > > > > > but
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > "the Catalog instance will be initialized if it
> has
> > > not
> > > > > > been
> > > > > > > > > > > > initialized
> > > > > > > > > > > > > > yet. If the initialization process fails, these
> > > > > statements
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > be
> > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > It looks more like lazy initialization for
> catalog
> > > than
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence
> of
> > > > > Catalog
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone, Thank you all for your interest in
> > this
> > > > > > DISCUSS.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > How to handle a catalog with the same name that
> > > > exists
> > > > > > for
> > > > > > > > both
> > > > > > > > > > of
> > > > > > > > > > > > > them?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I believe this is a crucial point. Based on my
> > > current
> > > > > > > personal
> > > > > > > > > > > > > > understanding, the Map<String, Catalog> catalogs
> > will
> > > > > serve
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > > cache
> > > > > > > > > > > > > > for instantiated catalogs and have the highest
> > > > priority.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There are three methods that can affect the
> > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. registerCatalog(String catalogName, Catalog
> > > catalog)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This method puts the catalog instance into the
> > > > > Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This method removes the catalog instance
> > > corresponding
> > > > to
> > > > > > > > > > catalogName
> > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This method first retrieves the corresponding
> > catalog
> > > > > > > instance
> > > > > > > > > from
> > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If the catalog
> > > does
> > > > > not
> > > > > > > > exist,
> > > > > > > > > > it
> > > > > > > > > > > > > > retrieves the corresponding configuration from
> the
> > > > > > > > CatalogStore,
> > > > > > > > > > > > > > initializes it, and puts the initialized Catalog
> > > > instance
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The following two methods only modify the
> > > configuration
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. addCatalog(String catalogName, Map<String,
> > String>
> > > > > > > > properties)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This method saves the properties to the
> > catalogStore
> > > > and
> > > > > > > checks
> > > > > > > > > > > > > > whether there is a catalogName with the same
> name.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > This method removes the specified configuration
> of
> > > the
> > > > > > > > specified
> > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The following are possible conflict scenarios:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. When the corresponding catalogName already
> > exists
> > > in
> > > > > the
> > > > > > > > > > > > > > CatalogStore but not in the Map<String, Catalog>,
> > the
> > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > catalog)
> > > > > method
> > > > > > > can
> > > > > > > > > > > > > > succeed and be directly saved to the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. When the corresponding catalogName already
> > exists
> > > in
> > > > > > both
> > > > > > > > the
> > > > > > > > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. When the corresponding catalogName already
> > exists
> > > in
> > > > > the
> > > > > > > > > > > > > > Map<String, Catalog>, the addCatalog(String
> > > > catalogName,
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > String> properties) method can directly save the
> > > > > properties
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > catalogName)
> > > > > method
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > > > use the new properties for initialization because
> > the
> > > > > > > > > corresponding
> > > > > > > > > > > > > > catalog instance already exists in catalogs and
> > will
> > > be
> > > > > > > > > > prioritized.
> > > > > > > > > > > > > > Therefore, using the unregisterCatalog(String
> > > > > catalogName)
> > > > > > > > method
> > > > > > > > > > to
> > > > > > > > > > > > > > remove the instance corresponding to the original
> > > > > > catalogName
> > > > > > > > is
> > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > `registerCatalog(String
> > > > > > > > > > > > > > catalogName,Catalog catalog)` in the `Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs`
> > > > > > > > > > > > and
> > > > > > > > > > > > > > `registerCatalog(String catalogName, Map<String,
> > > > String>
> > > > > > > > > > properties)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This could potentially lead to confusion. I
> suggest
> > > > > > changing
> > > > > > > > the
> > > > > > > > > > > > > > method name, perhaps to addCatalog(String
> > > catalogName,
> > > > > > > > > Map<String,
> > > > > > > > > > > > > > String> properties), as previously mentioned
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > add `registerCatalog(String catalogName,Catalog
> > > > > catalog,
> > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Since a catalog is already an instance, adding
> the
> > > > > > "lazyInit"
> > > > > > > > > > > > > > parameter to the registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > catalog)
> > > > > > > > > > > > > > method may not necessarily result in lazy
> > > > initialization.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think encryption is necessary, but perhaps the
> > > > > encryption
> > > > > > > > logic
> > > > > > > > > > > > > > should be implemented in the CatalogStore when
> the
> > > data
> > > > > is
> > > > > > > > > actually
> > > > > > > > > > > > > > saved. For instance, the FileCatalogStore could
> > > encrypt
> > > > > the
> > > > > > > > data
> > > > > > > > > > when
> > > > > > > > > > > > > > it is saved to a file, while the
> MemoryCatalogStore
> > > > does
> > > > > > not
> > > > > > > > > > require
> > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it is necessary to have a
> > MemoryCatalogStore
> > > as
> > > > > the
> > > > > > > > > default
> > > > > > > > > > > > > > implementation, which saves Catalog
> configurations
> > in
> > > > > > memory.
> > > > > > > > > > > > > > Otherwise, if we want to implement asynchronous
> > > loading
> > > > > of
> > > > > > > > > Catalog,
> > > > > > > > > > > we
> > > > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is also a critical concern. When executing
> > DDL,
> > > > > DQL,
> > > > > > or
> > > > > > > > DML
> > > > > > > > > > > > > > statements that reference a specific Catalog, the
> > > > Catalog
> > > > > > > > > instance
> > > > > > > > > > > > > > will be initialized if it has not been
> initialized
> > > yet.
> > > > > If
> > > > > > > the
> > > > > > > > > > > > > > initialization process fails, these statements
> will
> > > not
> > > > > be
> > > > > > > > > executed
> > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs and
> > > > CatalogStore
> > > > > > > > > > catalogstore
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the CatalogManager all have the same catalog
> name,
> > > but
> > > > > > > > > correponding
> > > > > > > > > > > to
> > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This issue can be resolved by referring to the
> > > previous
> > > > > > > > > responses.
> > > > > > > > > > > The
> > > > > > > > > > > > > > key principle is that the Catalog that has been
> > most
> > > > > > recently
> > > > > > > > > used
> > > > > > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for driving this Flip, I do believe this
> > > Flip
> > > > > > could
> > > > > > > be
> > > > > > > > > > > helpful
> > > > > > > > > > > > > > for users. The firm I work for also manages a lot
> > of
> > > > > > > catalogs,
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > > submission of tasks becomes slow because of
> > loading a
> > > > > > number
> > > > > > > of
> > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > We obtain the catalogs in the user's Flink SQL
> > > through
> > > > > > > regular
> > > > > > > > > > > > expression
> > > > > > > > > > > > > > to avoid loading all catalogs to improve the
> speed
> > of
> > > > > task
> > > > > > > > > > > submission.
> > > > > > > > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. When creating a catalog with CREATE CATALOG,
> > the
> > > > > > > > > asynchronous
> > > > > > > > > > > > > > registration method is used by default. What if
> > > > > > asynchronous
> > > > > > > > > > > > registration
> > > > > > > > > > > > > > failed? And how to implement asynchronous
> > > registration?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. I also have the same question with Shammon’s
> > > first
> > > > > > > > comment.
> > > > > > > > > > The
> > > > > > > > > > > > > > Map<String, Catalog> catalogs and CatalogStore
> > > > > catalogstore
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > CatalogManager all have the same catalog name,
> but
> > > > > > > correponding
> > > > > > > > > to
> > > > > > > > > > > > > > different catalog instances. For example:
> > catalog1->
> > > > > > > > jdbcCatalog
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the
> > > > > catalogstore.
> > > > > > > > Would
> > > > > > > > > > this
> > > > > > > > > > > > > case
> > > > > > > > > > > > > > happen? Maybe we should define `catalogs` and
> > > > > > `catalogstore`
> > > > > > > > more
> > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > ruanhang1993@gmail.com
> > > > >
> > > > > > 写道：
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi all, I would like to update you on the latest progress of the FLIP.
Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang and I
had an offline discussion regarding the overall solution for Flink
CatalogStore. We have reached a consensus and I have updated the final
solution in FLIP.
Next, let me briefly describe the entire design:
1.
Introduce CatalogDescriptor to store catalog configuration similar to
TableDescriptor.
2.
The two key functions of CatalogStore - void storeCatalog(String
catalogName, CatalogDescriptor) and CatalogDescriptor getCatalog(String)
will both use CatalogDescriptor instead of Catalog instance. This way,
CatalogStore will only be responsible for saving and retrieving catalog
configurations without having to initialize catalogs.
3.
The default registerCatalog(String catalogName, Catalog catalog)
function in CatalogManager will be marked as deprecated.
4.
A new function registerCatalog(String catalogName, CatalogDescriptor
catalog) will be added to serve as the default registration function for
catalogs in CatalogManager.
5.
Map<String,Catalog> catalogs in CataloManager will remain unchanged and
save initialized catalogs.This means that deletion operations from one
session won't synchronize with other sessions.
6.
To support multi-session synchronization scenarios for deletions later
on we should make Map<String,Catalog>catalogs configurable.There may be
three possible situations:
a.Default caching of all initialized catalogs
b.Introduction of LRU cache logic which can automatically clear
long-unused catalogs.
c.No caching of any instances; each call to getCatalog creates a new
instance.
This is the document for discussion:
https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
This is the final proposal document:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
Thank you very much for your attention and suggestions on this FLIP. A
special thanks to Hang Ruan for his suggestions on the entire design and
organizing offline discussions.
If you have any further suggestions or feedback about this FLIP please feel
free to share.
Best,
Feng
On Sat, May 6, 2023 at 8:32 PM Jing Ge <ji...@ververica.com.invalid> wrote:
> Hi Feng,
>
> Thanks for improving the FLIP. It looks good to me. We could still
> reconsider in the future how to provide more common built-in cache
> functionality in CatalogManager so that not every CatalogSotre
> implementation has to take care of it.
>
> Best regards,
> Jing
>
> On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com> wrote:
>
> > Hi Jing,
> >
> > Thanks for your reply.
> >
> > > There might be more such issues. I would suggest you completely walk
> > through the FLIP again and fix those issues
> >
> > I am very sorry for my carelessness and at the same time, I greatly
> > appreciate your careful review. I have thoroughly checked the entire FLIP
> > and made corrections to these issues.
> >
> >
> > > If I am not mistaken, with the current FLIP design, CatalogManager
> > could work without Optional CatalogStore being configured.
> >
> > Yes, in the original design, CatalogStore was not necessary because
> > CatalogManager used Map<String, Catalog> catalogs to store catalog
> > instances.
> > However, this caused inconsistency issues. Therefore, I modified this
> part
> > of the design and removed Map<String, Catalog> catalogs from
> > CatalogManager.
> > At the same time, InMemoryCatalog will serve as the default CatalogStore
> > to save catalogs in memory and replace the functionality of
> > Map<String,Catalog>catalogs.
> > The previous plan that kept Map<String,Catalog>catalogs has been moved to
> > Rejected Alternatives.
> >
> >
> >
> > Best,
> > Feng
> >
> > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > Hi Feng,
> > >
> > > There are still many places contain inconsistent content, e.g.
> > >
> > > 1. "Asynchronous registration" is still used.
> > > 2. The java comment of the method registerCatalog(String catalogName,
> > > Catalog catalog) in CatalogManager does not tell what the method is
> > doing.
> > >
> > >
> > > There might be more such issues. I would suggest you completely walk
> > > through the FLIP again and fix those issues.
> > >
> > > About the inMemoryCatalogStore, do you mean that you will build the
> cache
> > > functionality in the CatalogStore? This is a very different design
> > concept
> > > from what the current FLIP described. If I am not mistaken, with the
> > > current FLIP design, CatalogManager could work without Optional
> > > CatalogStore being configured. That is the reason why I mentioned in
> the
> > > last email that the example code wrt the Optional CatalogStore is not
> > > correct.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <ji...@gmail.com>
> wrote:
> > >
> > > > Hi Jing
> > > >
> > > >
> > > > > There are still some NIT issues in the FLIP
> > > >
> > > > Thank you very much for the careful review. I have already made the
> > > > relevant changes.
> > > >
> > > >
> > > > > Speaking of the conflict issues in the multi-instance scenarios, I
> > am
> > > > not
> > > > sure if this is the intended behaviour
> > > >
> > > > Currently, there are conflicts in multiple scenarios with the current
> > > > design. I am thinking whether we should remove 'Map<String, Catalog>'
> > and
> > > > make Cache the default behavior of InMemoryCatalogStore. This way,
> > users
> > > > can implement their own CatalogStore to achieve multi-instance
> > > > non-conflicting scenarios. What do you think?
> > > >
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi Feng,
> > > > >
> > > > > Thanks for working on the FLIP. There are still some NIT issues in
> > the
> > > > FLIP
> > > > > like:
> > > > >
> > > > > 1. Optional<CatalogStore> catalogStore has been used as
> CatalogStore
> > > > > instead of Optional in the code example. It should be fine to use
> it
> > as
> > > > > pseudo code for now and update it after you submit the PR.
> > > > > 2. addCatalog(...) is still used somewhere in the rejected section
> > > which
> > > > > should be persistContext(...) to keep it consistent.
> > > > >
> > > > > Speaking of the conflict issues in the multi-instance scenarios, I
> am
> > > not
> > > > > sure if this is the intended behaviour. If Map<String, Catalog>
> > > catalogs
> > > > is
> > > > > used as a cache, it should be invalid, once the related catalog has
> > > been
> > > > > removed from the CatalogStore by another instance. Did I miss
> > > something?
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Jing,Shammon
> > > > > > Thanks for your reply.
> > > > > >
> > > > > > @Jing
> > > > > >
> > > > > > > How about persistCatalog()?
> > > > > > I think this is a good function name, I have updated it in the
> > > > > > documentation.
> > > > > >
> > > > > > > Some common cache features should be implemented
> > > > > > Thank you for the suggestion. If alternative 1 turns out to be
> more
> > > > > > appropriate later, I will improve this part of the content.
> > > > > >
> > > > > > > As the above discussion moves forward, the option 2 solution
> > looks
> > > > more
> > > > > > like a replacement of option 1
> > > > > > Yes, after discussing with Shammon offline, we think that
> solution
> > 2
> > > > > might
> > > > > > be more suitable and also avoid any inconsistency issues.
> > > > > >
> > > > > > > There are some inconsistent descriptions in the content. Would
> > you
> > > > > like
> > > > > > to clean them up?
> > > > > > I will do my best to improve the document and appreciate your
> > > > > suggestions.
> > > > > >
> > > > > >
> > > > > >
> > > > > > @Shammon
> > > > > > > can you put the unselected option in `Rejected Alternatives`
> > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best
> > > > > > Feng
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi Feng
> > > > > > >
> > > > > > > Thanks for your update.
> > > > > > >
> > > > > > > I found there are two options in `Proposed Changes`, can you
> put
> > > the
> > > > > > > unselected option in `Rejected Alternatives`? I think this may
> > help
> > > > us
> > > > > > > better understand your proposal
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Shammon FY
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > Thanks for raising this FLIP. I am still confused after
> > > completely
> > > > > > > reading
> > > > > > > > the thread with following questions:
> > > > > > > >
> > > > > > > > 1. Naming confusion - registerCatalog() and addCatalog() have
> > no
> > > > big
> > > > > > > > difference based on their names. One of them is responsible
> for
> > > > data
> > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > 2. As you mentioned that Map<String, Catalog> catalogs is
> used
> > > as a
> > > > > > cache
> > > > > > > > and catalogStore is used for data persistence. I would
> suggest
> > > > > > describing
> > > > > > > > their purpose conceptually and clearly in the FLIP. Some
> common
> > > > cache
> > > > > > > > features should be implemented, i.e. data in the cache and
> the
> > > > store
> > > > > > > should
> > > > > > > > be consistent. Same Catalog instance should be found in the
> > store
> > > > and
> > > > > > in
> > > > > > > > the cache(either it has been initialized or it will be lazy
> > > > > > initialized)
> > > > > > > > for the same catalog name. The consistency will be taken care
> > of
> > > > > while
> > > > > > > > updating the catalog.
> > > > > > > > 3. As the above discussion moves forward, the option 2
> solution
> > > > looks
> > > > > > > more
> > > > > > > > like a replacement of option 1, because, afaiu, issues
> > mentioned
> > > > > > > > previously with option 1 are not solved yet. Do you still
> want
> > to
> > > > > > propose
> > > > > > > > both options and ask for suggestions for both of them?
> > > > > > > > 4. After you updated the FLIP, there are some inconsistent
> > > > > descriptions
> > > > > > > in
> > > > > > > > the content. Would you like to clean them up? Thanks!
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> jinfeng1995@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > hi Shammon
> > > > > > > > >
> > > > > > > > > Thank you for your response, and I completely agree with
> your
> > > > point
> > > > > > of
> > > > > > > > > view.
> > > > > > > > > Initially, I may have over complicated the whole issue.
> First
> > > and
> > > > > > > > foremost,
> > > > > > > > > we need to consider the persistence of the Catalog's
> > > > Configuration.
> > > > > > > > > If we only need to provide persistence for Catalog
> > > Configuration,
> > > > > we
> > > > > > > can
> > > > > > > > > add a toConfiguration method to the Catalog interface.
> > > > > > > > > This method can convert a Catalog instance to a Map<String,
> > > > String>
> > > > > > > > > properties, and the default implementation will throw an
> > > > exception.
> > > > > > > > >
> > > > > > > > > public interface Catalog {
> > > > > > > > > /**
> > > > > > > > > * Returns a map containing the properties of the catalog
> > > > object.
> > > > > > > > > *
> > > > > > > > > * @return a map containing the properties of the catalog
> > > > object
> > > > > > > > > * @throws UnsupportedOperationException if the
> > implementing
> > > > > class
> > > > > > > does
> > > > > > > > > not override
> > > > > > > > > * the default implementation of this method
> > > > > > > > > */
> > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > throw new UnsupportedOperationException("Please
> implement
> > > > > > > > toProperties
> > > > > > > > > for this catalog");
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > The specific process is as follows:
> > > > > > > > >
> > > > > > > > > 1. If the user has configured a CatalogStore, the
> > > > > toConfiguration()
> > > > > > > > method
> > > > > > > > > will be called when registering a Catalog instance with
> > > > > > > > > registerCatalog(String catalogName, Catalog catalog). The
> > > Catalog
> > > > > > > > instance
> > > > > > > > > will be converted to a Map<String, String> properties and
> > saved
> > > > to
> > > > > > the
> > > > > > > > > CatalogStore. If some Catalog instances do not implement
> this
> > > > > method,
> > > > > > > an
> > > > > > > > > exception will be thrown.
> > > > > > > > >
> > > > > > > > > 2. If the user does not use a CatalogStore, the
> > > > toConfiguration()
> > > > > > > method
> > > > > > > > > will not be called, ensuring consistency with the original
> > > > process.
> > > > > > > > >
> > > > > > > > > 3. Saving both the Map<String, Catalog> catalogs and the
> > > > > CatalogStore
> > > > > > > at
> > > > > > > > > the same time can also avoid conflicts
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > For lazy initialization:
> > > > > > > > > we can start from the Catalog itself and provide a
> dedicated
> > > > > Catalog
> > > > > > > > > implementation for lazy loading, such as
> ConfigurationCatalog
> > > > > > > > >
> > > > > > > > > public class ConfigurationCatalog implements Catalog {
> > > > > > > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > I have added this design to the FLIP.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > zjureel@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng
> > > > > > > > > >
> > > > > > > > > > Thanks for your answer.
> > > > > > > > > >
> > > > > > > > > > I have no questions about the functionality of
> > > `CatalogStore`,
> > > > > but
> > > > > > > the
> > > > > > > > > > different operations of multiple `registerCatalog` or
> > > > > > `storeCatalog`
> > > > > > > in
> > > > > > > > > > `CatalogManager`. The implementation in Trino is also the
> > > same,
> > > > > the
> > > > > > > > > > `CatalogManager` in trino only has one `createCatalog`,
> > which
> > > > > will
> > > > > > > save
> > > > > > > > > the
> > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > >
> > > > > > > > > > I think we don't need to add another `registerCatalog` or
> > > > > > > `addCatalog`
> > > > > > > > to
> > > > > > > > > > save a catalog in `Map<String, Catalog> catalogs` or
> > > > > `CatalogStore
> > > > > > > > > > catalogStore`. As you mentioned before, the `Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs` is a cache in `CatalogManager`. How about
> saving
> > > the
> > > > > > > catalog
> > > > > > > > in
> > > > > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore
> > > catalogStore`
> > > > > > > > together
> > > > > > > > > > when it is registered or added in `CatalogManager`?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shammon FY
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > jinfeng1995@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thank you for your reply. I am very sorry for the
> > > > > > misunderstanding
> > > > > > > > > caused
> > > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > > >
> > > > > > > > > > > @Shammon
> > > > > > > > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > > > > > > > Yes, there was indeed such a discussion before.
> However,
> > > in
> > > > > > > > designing
> > > > > > > > > > the
> > > > > > > > > > > whole solution, I found that the logic of
> CatalogManager
> > > > itself
> > > > > > > > doesn't
> > > > > > > > > > > need to change much. *We cannot only persist Catalog
> > > > instances
> > > > > > > > > > themselves*,
> > > > > > > > > > > so exposing only registerCatalog(String catalogName,
> > > Catalog
> > > > > > > catalog)
> > > > > > > > > > might
> > > > > > > > > > > not be enough to save Catalogs, because in the end we
> > still
> > > > > need
> > > > > > to
> > > > > > > > > save
> > > > > > > > > > > the configurations corresponding to the Catalog
> > instances.
> > > > > > > > Therefore,
> > > > > > > > > I
> > > > > > > > > > > decided to introduce the CatalogStore interface for
> > > > > configuration
> > > > > > > > > > > persistence. Regarding this part, I also took
> inspiration
> > > > from
> > > > > > > > Trino's
> > > > > > > > > > > implementation[1].
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > @yuxia
> > > > > > > > > > >
> > > > > > > > > > > > 1: The mechanism of handling catalog with the same
> name
> > > > > looks a
> > > > > > > > > little
> > > > > > > > > > of
> > > > > > > > > > > complex to me.
> > > > > > > > > > > Thank you for the suggestion. I will provide a detailed
> > > > > > description
> > > > > > > > and
> > > > > > > > > > > code examples for this part, and add it to the FLIP.
> > > > > > > > > > >
> > > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> > confused
> > > > to
> > > > > > me.
> > > > > > > > > IIUC,
> > > > > > > > > > > this method is for storing catalog to CatalogStore, how
> > > about
> > > > > > > > renaming
> > > > > > > > > it
> > > > > > > > > > > to `storeCatalog`? It's very personal opinion, you can
> > > decide
> > > > > to
> > > > > > > take
> > > > > > > > > it
> > > > > > > > > > or
> > > > > > > > > > > not by your self.
> > > > > > > > > > > StoreCatalog looks more intuitive to me. I don't see
> any
> > > > > problem
> > > > > > > with
> > > > > > > > > it.
> > > > > > > > > > >
> > > > > > > > > > > > 3.For CREATE CATALOG statement, which method will be
> > > > called?
> > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> whether
> > > user
> > > > > can
> > > > > > > > add a
> > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > For CREATE CATALOG, my original design was to add it
> > > directly
> > > > > to
> > > > > > > the
> > > > > > > > > > > CatalogStore, but this would disrupt the existing
> logic.
> > > > > > > Therefore, I
> > > > > > > > > > think
> > > > > > > > > > > we can do both: save the configuration to the
> > CatalogStore
> > > > and
> > > > > > > > > initialize
> > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > >
> > > > > > > > > > > > 3. Is it really neccessary to provide a default
> > > > implmentation
> > > > > > for
> > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > I think it is necessary, otherwise we would need to
> > > introduce
> > > > > an
> > > > > > > > > > additional
> > > > > > > > > > > Map to store the configuration for lazy loading.
> > > > > > > > > > >
> > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > I don't think registerCatalog(String catalogName,
> Catalog
> > > > > > catalog)
> > > > > > > > can
> > > > > > > > > be
> > > > > > > > > > > made into an asynchronous interface because Catalog is
> > > > already
> > > > > an
> > > > > > > > > > instance.
> > > > > > > > > > >
> > > > > > > > > > > > It looks more like lazy initialization for catalog
> than
> > > > > > > > asynchronous
> > > > > > > > > > > registration, right?
> > > > > > > > > > > Indeed, my description was inaccurate. It should be
> lazy
> > > > > > > registration
> > > > > > > > > > > instead of asynchronous registration. I have already
> > > updated
> > > > > the
> > > > > > > > title
> > > > > > > > > of
> > > > > > > > > > > the FLIP.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [1].
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > zjureel@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng
> > > > > > > > > > > >
> > > > > > > > > > > > I think if there is a `registerCatalog` method in
> > > > > > > `CatalogManager`,
> > > > > > > > > it
> > > > > > > > > > > will
> > > > > > > > > > > > confuse users whether a method named `addCatalog` or
> > > > > > > `storeCatalog`
> > > > > > > > > is
> > > > > > > > > > > > added.
> > > > > > > > > > > >
> > > > > > > > > > > > And as you mentioned, the memory catalog is a
> `cache`,
> > I
> > > > > think
> > > > > > > the
> > > > > > > > > > > concept
> > > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > > >
> > > > > > > > > > > > I found there is a pre-discussion [1] for this FLIP.
> > > Please
> > > > > > > correct
> > > > > > > > > me
> > > > > > > > > > if
> > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that discussion is
> > to
> > > > use
> > > > > > > > > > > > `CatalogManager` as an interface and implement it for
> > > > > different
> > > > > > > > > stores
> > > > > > > > > > > such
> > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > >
> > > > > > > > > > > > I think there is a gap between the current FLIP
> design
> > > and
> > > > > that
> > > > > > > > > > > conclusion.
> > > > > > > > > > > > What about the proposal of the discussion in thread
> > [1] ?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > >
> > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Shammon FY
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > 1: The mechanism of handling catalog with the same
> > name
> > > > > > looks a
> > > > > > > > > > little
> > > > > > > > > > > of
> > > > > > > > > > > > > complex to me. I think it'll be better to explain
> it
> > in
> > > > the
> > > > > > > java
> > > > > > > > > doc
> > > > > > > > > > of
> > > > > > > > > > > > > these methods and give a brief example in this
> FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> > > confused
> > > > > to
> > > > > > > me.
> > > > > > > > > > IIUC,
> > > > > > > > > > > > > this method is for storing catalog to CatalogStore,
> > how
> > > > > about
> > > > > > > > > > renaming
> > > > > > > > > > > it
> > > > > > > > > > > > > to `storeCatalog`? It's very personal opinion, you
> > can
> > > > > decide
> > > > > > > to
> > > > > > > > > take
> > > > > > > > > > > it
> > > > > > > > > > > > or
> > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3: For CREATE CATALOG statement, which method will
> be
> > > > > called?
> > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> > > whether
> > > > > user
> > > > > > > can
> > > > > > > > > > add a
> > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3: Is it really neccessary to provide a default
> > > > > implmentation
> > > > > > > for
> > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > When creating a catalog with CREATE CATALOG, the
> > > > > > asynchronous
> > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > If asynchronous registration is the default
> behavior,
> > > it
> > > > > > there
> > > > > > > > any
> > > > > > > > > > way
> > > > > > > > > > > > > that user can switch to synchronous registration
> just
> > > > like
> > > > > > > > before?
> > > > > > > > > > > > > Will both method `addCatalog` and `registerCatalog`
> > be
> > > > > > > > asynchronous
> > > > > > > > > > > > > registration?
> > > > > > > > > > > > >
> > > > > > > > > > > > > IIUC, in asynchronous registration, it may well
> that
> > > > CREATE
> > > > > > > > CATALOG
> > > > > > > > > > > > > executes successfully, but then the following
> CREATE
> > > > TABLE
> > > > > > > > > statement
> > > > > > > > > > > will
> > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > I think it's a break change which should be
> > highlighted
> > > > in
> > > > > > this
> > > > > > > > > FLIP,
> > > > > > > > > > > may
> > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > BTW, by saying asynchronous registration, I would
> > like
> > > to
> > > > > > > expect
> > > > > > > > > > there
> > > > > > > > > > > > > will be an executor to open or register catalog in
> > the
> > > > > > > > background,
> > > > > > > > > > but
> > > > > > > > > > > > from
> > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > "the Catalog instance will be initialized if it has
> > not
> > > > > been
> > > > > > > > > > > initialized
> > > > > > > > > > > > > yet. If the initialization process fails, these
> > > > statements
> > > > > > will
> > > > > > > > not
> > > > > > > > > > be
> > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > It looks more like lazy initialization for catalog
> > than
> > > > > > > > > asynchronous
> > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > >
> > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > > > Catalog
> > > > > > > > > > > configuration
> > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi everyone, Thank you all for your interest in
> this
> > > > > DISCUSS.
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > How to handle a catalog with the same name that
> > > exists
> > > > > for
> > > > > > > both
> > > > > > > > > of
> > > > > > > > > > > > them?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I believe this is a crucial point. Based on my
> > current
> > > > > > personal
> > > > > > > > > > > > > understanding, the Map<String, Catalog> catalogs
> will
> > > > serve
> > > > > > as
> > > > > > > a
> > > > > > > > > > cache
> > > > > > > > > > > > > for instantiated catalogs and have the highest
> > > priority.
> > > > > > > > > > > > >
> > > > > > > > > > > > > There are three methods that can affect the
> > Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. registerCatalog(String catalogName, Catalog
> > catalog)
> > > > > > > > > > > > >
> > > > > > > > > > > > > This method puts the catalog instance into the
> > > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > > >
> > > > > > > > > > > > > This method removes the catalog instance
> > corresponding
> > > to
> > > > > > > > > catalogName
> > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > >
> > > > > > > > > > > > > This method first retrieves the corresponding
> catalog
> > > > > > instance
> > > > > > > > from
> > > > > > > > > > > > > the Map<String, Catalog> catalogs. If the catalog
> > does
> > > > not
> > > > > > > exist,
> > > > > > > > > it
> > > > > > > > > > > > > retrieves the corresponding configuration from the
> > > > > > > CatalogStore,
> > > > > > > > > > > > > initializes it, and puts the initialized Catalog
> > > instance
> > > > > > into
> > > > > > > > the
> > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The following two methods only modify the
> > configuration
> > > > in
> > > > > > the
> > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. addCatalog(String catalogName, Map<String,
> String>
> > > > > > > properties)
> > > > > > > > > > > > >
> > > > > > > > > > > > > This method saves the properties to the
> catalogStore
> > > and
> > > > > > checks
> > > > > > > > > > > > > whether there is a catalogName with the same name.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > This method removes the specified configuration of
> > the
> > > > > > > specified
> > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The following are possible conflict scenarios:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. When the corresponding catalogName already
> exists
> > in
> > > > the
> > > > > > > > > > > > > CatalogStore but not in the Map<String, Catalog>,
> the
> > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> catalog)
> > > > method
> > > > > > can
> > > > > > > > > > > > > succeed and be directly saved to the Map<String,
> > > Catalog>
> > > > > > > > catalogs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. When the corresponding catalogName already
> exists
> > in
> > > > > both
> > > > > > > the
> > > > > > > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. When the corresponding catalogName already
> exists
> > in
> > > > the
> > > > > > > > > > > > > Map<String, Catalog>, the addCatalog(String
> > > catalogName,
> > > > > > > > > Map<String,
> > > > > > > > > > > > > String> properties) method can directly save the
> > > > properties
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > catalogStore, but the getCatalog(String
> catalogName)
> > > > method
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > > > use the new properties for initialization because
> the
> > > > > > > > corresponding
> > > > > > > > > > > > > catalog instance already exists in catalogs and
> will
> > be
> > > > > > > > > prioritized.
> > > > > > > > > > > > > Therefore, using the unregisterCatalog(String
> > > > catalogName)
> > > > > > > method
> > > > > > > > > to
> > > > > > > > > > > > > remove the instance corresponding to the original
> > > > > catalogName
> > > > > > > is
> > > > > > > > > > > > > necessary.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it will confuse users that
> > > > > `registerCatalog(String
> > > > > > > > > > > > > catalogName,Catalog catalog)` in the `Map<String,
> > > > Catalog>
> > > > > > > > > catalogs`
> > > > > > > > > > > and
> > > > > > > > > > > > > `registerCatalog(String catalogName, Map<String,
> > > String>
> > > > > > > > > properties)
> > > > > > > > > > > > >
> > > > > > > > > > > > > This could potentially lead to confusion. I suggest
> > > > > changing
> > > > > > > the
> > > > > > > > > > > > > method name, perhaps to addCatalog(String
> > catalogName,
> > > > > > > > Map<String,
> > > > > > > > > > > > > String> properties), as previously mentioned
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > add `registerCatalog(String catalogName,Catalog
> > > > catalog,
> > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > >
> > > > > > > > > > > > > Since a catalog is already an instance, adding the
> > > > > "lazyInit"
> > > > > > > > > > > > > parameter to the registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > > > method may not necessarily result in lazy
> > > initialization.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think encryption is necessary, but perhaps the
> > > > encryption
> > > > > > > logic
> > > > > > > > > > > > > should be implemented in the CatalogStore when the
> > data
> > > > is
> > > > > > > > actually
> > > > > > > > > > > > > saved. For instance, the FileCatalogStore could
> > encrypt
> > > > the
> > > > > > > data
> > > > > > > > > when
> > > > > > > > > > > > > it is saved to a file, while the MemoryCatalogStore
> > > does
> > > > > not
> > > > > > > > > require
> > > > > > > > > > > > > encryption.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think it is necessary to have a
> MemoryCatalogStore
> > as
> > > > the
> > > > > > > > default
> > > > > > > > > > > > > implementation, which saves Catalog configurations
> in
> > > > > memory.
> > > > > > > > > > > > > Otherwise, if we want to implement asynchronous
> > loading
> > > > of
> > > > > > > > Catalog,
> > > > > > > > > > we
> > > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > >
> > > > > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is also a critical concern. When executing
> DDL,
> > > > DQL,
> > > > > or
> > > > > > > DML
> > > > > > > > > > > > > statements that reference a specific Catalog, the
> > > Catalog
> > > > > > > > instance
> > > > > > > > > > > > > will be initialized if it has not been initialized
> > yet.
> > > > If
> > > > > > the
> > > > > > > > > > > > > initialization process fails, these statements will
> > not
> > > > be
> > > > > > > > executed
> > > > > > > > > > > > > successfully.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs and
> > > CatalogStore
> > > > > > > > > catalogstore
> > > > > > > > > > > in
> > > > > > > > > > > > > the CatalogManager all have the same catalog name,
> > but
> > > > > > > > correponding
> > > > > > > > > > to
> > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This issue can be resolved by referring to the
> > previous
> > > > > > > > responses.
> > > > > > > > > > The
> > > > > > > > > > > > > key principle is that the Catalog that has been
> most
> > > > > recently
> > > > > > > > used
> > > > > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for driving this Flip, I do believe this
> > Flip
> > > > > could
> > > > > > be
> > > > > > > > > > helpful
> > > > > > > > > > > > > for users. The firm I work for also manages a lot
> of
> > > > > > catalogs,
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > submission of tasks becomes slow because of
> loading a
> > > > > number
> > > > > > of
> > > > > > > > > > > catalogs.
> > > > > > > > > > > > > We obtain the catalogs in the user's Flink SQL
> > through
> > > > > > regular
> > > > > > > > > > > expression
> > > > > > > > > > > > > to avoid loading all catalogs to improve the speed
> of
> > > > task
> > > > > > > > > > submission.
> > > > > > > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. When creating a catalog with CREATE CATALOG,
> the
> > > > > > > > asynchronous
> > > > > > > > > > > > > registration method is used by default. What if
> > > > > asynchronous
> > > > > > > > > > > registration
> > > > > > > > > > > > > failed? And how to implement asynchronous
> > registration?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. I also have the same question with Shammon’s
> > first
> > > > > > > comment.
> > > > > > > > > The
> > > > > > > > > > > > > Map<String, Catalog> catalogs and CatalogStore
> > > > catalogstore
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > CatalogManager all have the same catalog name, but
> > > > > > correponding
> > > > > > > > to
> > > > > > > > > > > > > different catalog instances. For example:
> catalog1->
> > > > > > > jdbcCatalog
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the
> > > > catalogstore.
> > > > > > > Would
> > > > > > > > > this
> > > > > > > > > > > > case
> > > > > > > > > > > > > happen? Maybe we should define `catalogs` and
> > > > > `catalogstore`
> > > > > > > more
> > > > > > > > > > > > clearly.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > ruanhang1993@gmail.com
> > > >
> > > > > 写道：
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Feng,
Thanks for improving the FLIP. It looks good to me. We could still
reconsider in the future how to provide more common built-in cache
functionality in CatalogManager so that not every CatalogSotre
implementation has to take care of it.
Best regards,
Jing
On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com> wrote:
> Hi Jing,
>
> Thanks for your reply.
>
> > There might be more such issues. I would suggest you completely walk
> through the FLIP again and fix those issues
>
> I am very sorry for my carelessness and at the same time, I greatly
> appreciate your careful review. I have thoroughly checked the entire FLIP
> and made corrections to these issues.
>
>
> > If I am not mistaken, with the current FLIP design, CatalogManager
> could work without Optional CatalogStore being configured.
>
> Yes, in the original design, CatalogStore was not necessary because
> CatalogManager used Map<String, Catalog> catalogs to store catalog
> instances.
> However, this caused inconsistency issues. Therefore, I modified this part
> of the design and removed Map<String, Catalog> catalogs from
> CatalogManager.
> At the same time, InMemoryCatalog will serve as the default CatalogStore
> to save catalogs in memory and replace the functionality of
> Map<String,Catalog>catalogs.
> The previous plan that kept Map<String,Catalog>catalogs has been moved to
> Rejected Alternatives.
>
>
>
> Best,
> Feng
>
> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi Feng,
> >
> > There are still many places contain inconsistent content, e.g.
> >
> > 1. "Asynchronous registration" is still used.
> > 2. The java comment of the method registerCatalog(String catalogName,
> > Catalog catalog) in CatalogManager does not tell what the method is
> doing.
> >
> >
> > There might be more such issues. I would suggest you completely walk
> > through the FLIP again and fix those issues.
> >
> > About the inMemoryCatalogStore, do you mean that you will build the cache
> > functionality in the CatalogStore? This is a very different design
> concept
> > from what the current FLIP described. If I am not mistaken, with the
> > current FLIP design, CatalogManager could work without Optional
> > CatalogStore being configured. That is the reason why I mentioned in the
> > last email that the example code wrt the Optional CatalogStore is not
> > correct.
> >
> > Best regards,
> > Jing
> >
> > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <ji...@gmail.com> wrote:
> >
> > > Hi Jing
> > >
> > >
> > > > There are still some NIT issues in the FLIP
> > >
> > > Thank you very much for the careful review. I have already made the
> > > relevant changes.
> > >
> > >
> > > > Speaking of the conflict issues in the multi-instance scenarios, I
> am
> > > not
> > > sure if this is the intended behaviour
> > >
> > > Currently, there are conflicts in multiple scenarios with the current
> > > design. I am thinking whether we should remove 'Map<String, Catalog>'
> and
> > > make Cache the default behavior of InMemoryCatalogStore. This way,
> users
> > > can implement their own CatalogStore to achieve multi-instance
> > > non-conflicting scenarios. What do you think?
> > >
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > >
> > > > Hi Feng,
> > > >
> > > > Thanks for working on the FLIP. There are still some NIT issues in
> the
> > > FLIP
> > > > like:
> > > >
> > > > 1. Optional<CatalogStore> catalogStore has been used as CatalogStore
> > > > instead of Optional in the code example. It should be fine to use it
> as
> > > > pseudo code for now and update it after you submit the PR.
> > > > 2. addCatalog(...) is still used somewhere in the rejected section
> > which
> > > > should be persistContext(...) to keep it consistent.
> > > >
> > > > Speaking of the conflict issues in the multi-instance scenarios, I am
> > not
> > > > sure if this is the intended behaviour. If Map<String, Catalog>
> > catalogs
> > > is
> > > > used as a cache, it should be invalid, once the related catalog has
> > been
> > > > removed from the CatalogStore by another instance. Did I miss
> > something?
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <ji...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Jing,Shammon
> > > > > Thanks for your reply.
> > > > >
> > > > > @Jing
> > > > >
> > > > > > How about persistCatalog()?
> > > > > I think this is a good function name, I have updated it in the
> > > > > documentation.
> > > > >
> > > > > > Some common cache features should be implemented
> > > > > Thank you for the suggestion. If alternative 1 turns out to be more
> > > > > appropriate later, I will improve this part of the content.
> > > > >
> > > > > > As the above discussion moves forward, the option 2 solution
> looks
> > > more
> > > > > like a replacement of option 1
> > > > > Yes, after discussing with Shammon offline, we think that solution
> 2
> > > > might
> > > > > be more suitable and also avoid any inconsistency issues.
> > > > >
> > > > > > There are some inconsistent descriptions in the content. Would
> you
> > > > like
> > > > > to clean them up?
> > > > > I will do my best to improve the document and appreciate your
> > > > suggestions.
> > > > >
> > > > >
> > > > >
> > > > > @Shammon
> > > > > > can you put the unselected option in `Rejected Alternatives`
> > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > >
> > > > >
> > > > >
> > > > > Best
> > > > > Feng
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Feng
> > > > > >
> > > > > > Thanks for your update.
> > > > > >
> > > > > > I found there are two options in `Proposed Changes`, can you put
> > the
> > > > > > unselected option in `Rejected Alternatives`? I think this may
> help
> > > us
> > > > > > better understand your proposal
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Shammon FY
> > > > > >
> > > > > >
> > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > Thanks for raising this FLIP. I am still confused after
> > completely
> > > > > > reading
> > > > > > > the thread with following questions:
> > > > > > >
> > > > > > > 1. Naming confusion - registerCatalog() and addCatalog() have
> no
> > > big
> > > > > > > difference based on their names. One of them is responsible for
> > > data
> > > > > > > persistence. How about persistCatalog()?
> > > > > > > 2. As you mentioned that Map<String, Catalog> catalogs is used
> > as a
> > > > > cache
> > > > > > > and catalogStore is used for data persistence. I would suggest
> > > > > describing
> > > > > > > their purpose conceptually and clearly in the FLIP. Some common
> > > cache
> > > > > > > features should be implemented, i.e. data in the cache and the
> > > store
> > > > > > should
> > > > > > > be consistent. Same Catalog instance should be found in the
> store
> > > and
> > > > > in
> > > > > > > the cache(either it has been initialized or it will be lazy
> > > > > initialized)
> > > > > > > for the same catalog name. The consistency will be taken care
> of
> > > > while
> > > > > > > updating the catalog.
> > > > > > > 3. As the above discussion moves forward, the option 2 solution
> > > looks
> > > > > > more
> > > > > > > like a replacement of option 1, because, afaiu, issues
> mentioned
> > > > > > > previously with option 1 are not solved yet. Do you still want
> to
> > > > > propose
> > > > > > > both options and ask for suggestions for both of them?
> > > > > > > 4. After you updated the FLIP, there are some inconsistent
> > > > descriptions
> > > > > > in
> > > > > > > the content. Would you like to clean them up? Thanks!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <jinfeng1995@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > hi Shammon
> > > > > > > >
> > > > > > > > Thank you for your response, and I completely agree with your
> > > point
> > > > > of
> > > > > > > > view.
> > > > > > > > Initially, I may have over complicated the whole issue. First
> > and
> > > > > > > foremost,
> > > > > > > > we need to consider the persistence of the Catalog's
> > > Configuration.
> > > > > > > > If we only need to provide persistence for Catalog
> > Configuration,
> > > > we
> > > > > > can
> > > > > > > > add a toConfiguration method to the Catalog interface.
> > > > > > > > This method can convert a Catalog instance to a Map<String,
> > > String>
> > > > > > > > properties, and the default implementation will throw an
> > > exception.
> > > > > > > >
> > > > > > > > public interface Catalog {
> > > > > > > > /**
> > > > > > > > * Returns a map containing the properties of the catalog
> > > object.
> > > > > > > > *
> > > > > > > > * @return a map containing the properties of the catalog
> > > object
> > > > > > > > * @throws UnsupportedOperationException if the
> implementing
> > > > class
> > > > > > does
> > > > > > > > not override
> > > > > > > > * the default implementation of this method
> > > > > > > > */
> > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > throw new UnsupportedOperationException("Please implement
> > > > > > > toProperties
> > > > > > > > for this catalog");
> > > > > > > > }
> > > > > > > > }
> > > > > > > >
> > > > > > > > The specific process is as follows:
> > > > > > > >
> > > > > > > > 1. If the user has configured a CatalogStore, the
> > > > toConfiguration()
> > > > > > > method
> > > > > > > > will be called when registering a Catalog instance with
> > > > > > > > registerCatalog(String catalogName, Catalog catalog). The
> > Catalog
> > > > > > > instance
> > > > > > > > will be converted to a Map<String, String> properties and
> saved
> > > to
> > > > > the
> > > > > > > > CatalogStore. If some Catalog instances do not implement this
> > > > method,
> > > > > > an
> > > > > > > > exception will be thrown.
> > > > > > > >
> > > > > > > > 2. If the user does not use a CatalogStore, the
> > > toConfiguration()
> > > > > > method
> > > > > > > > will not be called, ensuring consistency with the original
> > > process.
> > > > > > > >
> > > > > > > > 3. Saving both the Map<String, Catalog> catalogs and the
> > > > CatalogStore
> > > > > > at
> > > > > > > > the same time can also avoid conflicts
> > > > > > > >
> > > > > > > >
> > > > > > > > For lazy initialization:
> > > > > > > > we can start from the Catalog itself and provide a dedicated
> > > > Catalog
> > > > > > > > implementation for lazy loading, such as ConfigurationCatalog
> > > > > > > >
> > > > > > > > public class ConfigurationCatalog implements Catalog {
> > > > > > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > > > > > }
> > > > > > > > }
> > > > > > > >
> > > > > > > > I have added this design to the FLIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng
> > > > > > > >
> > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> zjureel@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng
> > > > > > > > >
> > > > > > > > > Thanks for your answer.
> > > > > > > > >
> > > > > > > > > I have no questions about the functionality of
> > `CatalogStore`,
> > > > but
> > > > > > the
> > > > > > > > > different operations of multiple `registerCatalog` or
> > > > > `storeCatalog`
> > > > > > in
> > > > > > > > > `CatalogManager`. The implementation in Trino is also the
> > same,
> > > > the
> > > > > > > > > `CatalogManager` in trino only has one `createCatalog`,
> which
> > > > will
> > > > > > save
> > > > > > > > the
> > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > >
> > > > > > > > > I think we don't need to add another `registerCatalog` or
> > > > > > `addCatalog`
> > > > > > > to
> > > > > > > > > save a catalog in `Map<String, Catalog> catalogs` or
> > > > `CatalogStore
> > > > > > > > > catalogStore`. As you mentioned before, the `Map<String,
> > > > Catalog>
> > > > > > > > > catalogs` is a cache in `CatalogManager`. How about saving
> > the
> > > > > > catalog
> > > > > > > in
> > > > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore
> > catalogStore`
> > > > > > > together
> > > > > > > > > when it is registered or added in `CatalogManager`?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shammon FY
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > jinfeng1995@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you for your reply. I am very sorry for the
> > > > > misunderstanding
> > > > > > > > caused
> > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > >
> > > > > > > > > > @Shammon
> > > > > > > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > > > > > > Yes, there was indeed such a discussion before. However,
> > in
> > > > > > > designing
> > > > > > > > > the
> > > > > > > > > > whole solution, I found that the logic of CatalogManager
> > > itself
> > > > > > > doesn't
> > > > > > > > > > need to change much. *We cannot only persist Catalog
> > > instances
> > > > > > > > > themselves*,
> > > > > > > > > > so exposing only registerCatalog(String catalogName,
> > Catalog
> > > > > > catalog)
> > > > > > > > > might
> > > > > > > > > > not be enough to save Catalogs, because in the end we
> still
> > > > need
> > > > > to
> > > > > > > > save
> > > > > > > > > > the configurations corresponding to the Catalog
> instances.
> > > > > > > Therefore,
> > > > > > > > I
> > > > > > > > > > decided to introduce the CatalogStore interface for
> > > > configuration
> > > > > > > > > > persistence. Regarding this part, I also took inspiration
> > > from
> > > > > > > Trino's
> > > > > > > > > > implementation[1].
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > @yuxia
> > > > > > > > > >
> > > > > > > > > > > 1: The mechanism of handling catalog with the same name
> > > > looks a
> > > > > > > > little
> > > > > > > > > of
> > > > > > > > > > complex to me.
> > > > > > > > > > Thank you for the suggestion. I will provide a detailed
> > > > > description
> > > > > > > and
> > > > > > > > > > code examples for this part, and add it to the FLIP.
> > > > > > > > > >
> > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> confused
> > > to
> > > > > me.
> > > > > > > > IIUC,
> > > > > > > > > > this method is for storing catalog to CatalogStore, how
> > about
> > > > > > > renaming
> > > > > > > > it
> > > > > > > > > > to `storeCatalog`? It's very personal opinion, you can
> > decide
> > > > to
> > > > > > take
> > > > > > > > it
> > > > > > > > > or
> > > > > > > > > > not by your self.
> > > > > > > > > > StoreCatalog looks more intuitive to me. I don't see any
> > > > problem
> > > > > > with
> > > > > > > > it.
> > > > > > > > > >
> > > > > > > > > > > 3.For CREATE CATALOG statement, which method will be
> > > called?
> > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether
> > user
> > > > can
> > > > > > > add a
> > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > For CREATE CATALOG, my original design was to add it
> > directly
> > > > to
> > > > > > the
> > > > > > > > > > CatalogStore, but this would disrupt the existing logic.
> > > > > > Therefore, I
> > > > > > > > > think
> > > > > > > > > > we can do both: save the configuration to the
> CatalogStore
> > > and
> > > > > > > > initialize
> > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > >
> > > > > > > > > > > 3. Is it really neccessary to provide a default
> > > implmentation
> > > > > for
> > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > I think it is necessary, otherwise we would need to
> > introduce
> > > > an
> > > > > > > > > additional
> > > > > > > > > > Map to store the configuration for lazy loading.
> > > > > > > > > >
> > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > I don't think registerCatalog(String catalogName, Catalog
> > > > > catalog)
> > > > > > > can
> > > > > > > > be
> > > > > > > > > > made into an asynchronous interface because Catalog is
> > > already
> > > > an
> > > > > > > > > instance.
> > > > > > > > > >
> > > > > > > > > > > It looks more like lazy initialization for catalog than
> > > > > > > asynchronous
> > > > > > > > > > registration, right?
> > > > > > > > > > Indeed, my description was inaccurate. It should be lazy
> > > > > > registration
> > > > > > > > > > instead of asynchronous registration. I have already
> > updated
> > > > the
> > > > > > > title
> > > > > > > > of
> > > > > > > > > > the FLIP.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [1].
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > zjureel@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng
> > > > > > > > > > >
> > > > > > > > > > > I think if there is a `registerCatalog` method in
> > > > > > `CatalogManager`,
> > > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > > confuse users whether a method named `addCatalog` or
> > > > > > `storeCatalog`
> > > > > > > > is
> > > > > > > > > > > added.
> > > > > > > > > > >
> > > > > > > > > > > And as you mentioned, the memory catalog is a `cache`,
> I
> > > > think
> > > > > > the
> > > > > > > > > > concept
> > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > >
> > > > > > > > > > > I found there is a pre-discussion [1] for this FLIP.
> > Please
> > > > > > correct
> > > > > > > > me
> > > > > > > > > if
> > > > > > > > > > > I'm wrong, IIUC, the conclusion of that discussion is
> to
> > > use
> > > > > > > > > > > `CatalogManager` as an interface and implement it for
> > > > different
> > > > > > > > stores
> > > > > > > > > > such
> > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > >
> > > > > > > > > > > I think there is a gap between the current FLIP design
> > and
> > > > that
> > > > > > > > > > conclusion.
> > > > > > > > > > > What about the proposal of the discussion in thread
> [1] ?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > >
> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Shammon FY
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > 1: The mechanism of handling catalog with the same
> name
> > > > > looks a
> > > > > > > > > little
> > > > > > > > > > of
> > > > > > > > > > > > complex to me. I think it'll be better to explain it
> in
> > > the
> > > > > > java
> > > > > > > > doc
> > > > > > > > > of
> > > > > > > > > > > > these methods and give a brief example in this FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> > confused
> > > > to
> > > > > > me.
> > > > > > > > > IIUC,
> > > > > > > > > > > > this method is for storing catalog to CatalogStore,
> how
> > > > about
> > > > > > > > > renaming
> > > > > > > > > > it
> > > > > > > > > > > > to `storeCatalog`? It's very personal opinion, you
> can
> > > > decide
> > > > > > to
> > > > > > > > take
> > > > > > > > > > it
> > > > > > > > > > > or
> > > > > > > > > > > > not by your self.
> > > > > > > > > > > >
> > > > > > > > > > > > 3: For CREATE CATALOG statement, which method will be
> > > > called?
> > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> > whether
> > > > user
> > > > > > can
> > > > > > > > > add a
> > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 3: Is it really neccessary to provide a default
> > > > implmentation
> > > > > > for
> > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > > >
> > > > > > > > > > > > > When creating a catalog with CREATE CATALOG, the
> > > > > asynchronous
> > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > If asynchronous registration is the default behavior,
> > it
> > > > > there
> > > > > > > any
> > > > > > > > > way
> > > > > > > > > > > > that user can switch to synchronous registration just
> > > like
> > > > > > > before?
> > > > > > > > > > > > Will both method `addCatalog` and `registerCatalog`
> be
> > > > > > > asynchronous
> > > > > > > > > > > > registration?
> > > > > > > > > > > >
> > > > > > > > > > > > IIUC, in asynchronous registration, it may well that
> > > CREATE
> > > > > > > CATALOG
> > > > > > > > > > > > executes successfully, but then the following CREATE
> > > TABLE
> > > > > > > > statement
> > > > > > > > > > will
> > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > I think it's a break change which should be
> highlighted
> > > in
> > > > > this
> > > > > > > > FLIP,
> > > > > > > > > > may
> > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > BTW, by saying asynchronous registration, I would
> like
> > to
> > > > > > expect
> > > > > > > > > there
> > > > > > > > > > > > will be an executor to open or register catalog in
> the
> > > > > > > background,
> > > > > > > > > but
> > > > > > > > > > > from
> > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > "the Catalog instance will be initialized if it has
> not
> > > > been
> > > > > > > > > > initialized
> > > > > > > > > > > > yet. If the initialization process fails, these
> > > statements
> > > > > will
> > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > It looks more like lazy initialization for catalog
> than
> > > > > > > > asynchronous
> > > > > > > > > > > > registration, right?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Yuxia
> > > > > > > > > > > >
> > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > > Catalog
> > > > > > > > > > configuration
> > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > >
> > > > > > > > > > > > Hi everyone, Thank you all for your interest in this
> > > > DISCUSS.
> > > > > > > > > > > >
> > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > How to handle a catalog with the same name that
> > exists
> > > > for
> > > > > > both
> > > > > > > > of
> > > > > > > > > > > them?
> > > > > > > > > > > >
> > > > > > > > > > > > I believe this is a crucial point. Based on my
> current
> > > > > personal
> > > > > > > > > > > > understanding, the Map<String, Catalog> catalogs will
> > > serve
> > > > > as
> > > > > > a
> > > > > > > > > cache
> > > > > > > > > > > > for instantiated catalogs and have the highest
> > priority.
> > > > > > > > > > > >
> > > > > > > > > > > > There are three methods that can affect the
> Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. registerCatalog(String catalogName, Catalog
> catalog)
> > > > > > > > > > > >
> > > > > > > > > > > > This method puts the catalog instance into the
> > > Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > >
> > > > > > > > > > > > This method removes the catalog instance
> corresponding
> > to
> > > > > > > > catalogName
> > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > >
> > > > > > > > > > > > This method first retrieves the corresponding catalog
> > > > > instance
> > > > > > > from
> > > > > > > > > > > > the Map<String, Catalog> catalogs. If the catalog
> does
> > > not
> > > > > > exist,
> > > > > > > > it
> > > > > > > > > > > > retrieves the corresponding configuration from the
> > > > > > CatalogStore,
> > > > > > > > > > > > initializes it, and puts the initialized Catalog
> > instance
> > > > > into
> > > > > > > the
> > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > >
> > > > > > > > > > > > The following two methods only modify the
> configuration
> > > in
> > > > > the
> > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. addCatalog(String catalogName, Map<String, String>
> > > > > > properties)
> > > > > > > > > > > >
> > > > > > > > > > > > This method saves the properties to the catalogStore
> > and
> > > > > checks
> > > > > > > > > > > > whether there is a catalogName with the same name.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > This method removes the specified configuration of
> the
> > > > > > specified
> > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > >
> > > > > > > > > > > > The following are possible conflict scenarios:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. When the corresponding catalogName already exists
> in
> > > the
> > > > > > > > > > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > > > > > > > > > registerCatalog(String catalogName, Catalog catalog)
> > > method
> > > > > can
> > > > > > > > > > > > succeed and be directly saved to the Map<String,
> > Catalog>
> > > > > > > catalogs.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. When the corresponding catalogName already exists
> in
> > > > both
> > > > > > the
> > > > > > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > > > > > registerCatalog(String
> > > > > > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. When the corresponding catalogName already exists
> in
> > > the
> > > > > > > > > > > > Map<String, Catalog>, the addCatalog(String
> > catalogName,
> > > > > > > > Map<String,
> > > > > > > > > > > > String> properties) method can directly save the
> > > properties
> > > > > to
> > > > > > > the
> > > > > > > > > > > > catalogStore, but the getCatalog(String catalogName)
> > > method
> > > > > > will
> > > > > > > > not
> > > > > > > > > > > > use the new properties for initialization because the
> > > > > > > corresponding
> > > > > > > > > > > > catalog instance already exists in catalogs and will
> be
> > > > > > > > prioritized.
> > > > > > > > > > > > Therefore, using the unregisterCatalog(String
> > > catalogName)
> > > > > > method
> > > > > > > > to
> > > > > > > > > > > > remove the instance corresponding to the original
> > > > catalogName
> > > > > > is
> > > > > > > > > > > > necessary.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > I think it will confuse users that
> > > > `registerCatalog(String
> > > > > > > > > > > > catalogName,Catalog catalog)` in the `Map<String,
> > > Catalog>
> > > > > > > > catalogs`
> > > > > > > > > > and
> > > > > > > > > > > > `registerCatalog(String catalogName, Map<String,
> > String>
> > > > > > > > properties)
> > > > > > > > > > > >
> > > > > > > > > > > > This could potentially lead to confusion. I suggest
> > > > changing
> > > > > > the
> > > > > > > > > > > > method name, perhaps to addCatalog(String
> catalogName,
> > > > > > > Map<String,
> > > > > > > > > > > > String> properties), as previously mentioned
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > @Hang
> > > > > > > > > > > > > add `registerCatalog(String catalogName,Catalog
> > > catalog,
> > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > >
> > > > > > > > > > > > Since a catalog is already an instance, adding the
> > > > "lazyInit"
> > > > > > > > > > > > parameter to the registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > > method may not necessarily result in lazy
> > initialization.
> > > > > > > > > > > >
> > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > >
> > > > > > > > > > > > I think encryption is necessary, but perhaps the
> > > encryption
> > > > > > logic
> > > > > > > > > > > > should be implemented in the CatalogStore when the
> data
> > > is
> > > > > > > actually
> > > > > > > > > > > > saved. For instance, the FileCatalogStore could
> encrypt
> > > the
> > > > > > data
> > > > > > > > when
> > > > > > > > > > > > it is saved to a file, while the MemoryCatalogStore
> > does
> > > > not
> > > > > > > > require
> > > > > > > > > > > > encryption.
> > > > > > > > > > > >
> > > > > > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > > > > > >
> > > > > > > > > > > > I think it is necessary to have a MemoryCatalogStore
> as
> > > the
> > > > > > > default
> > > > > > > > > > > > implementation, which saves Catalog configurations in
> > > > memory.
> > > > > > > > > > > > Otherwise, if we want to implement asynchronous
> loading
> > > of
> > > > > > > Catalog,
> > > > > > > > > we
> > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > @Xianxun
> > > > > > > > > > > >
> > > > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > > > >
> > > > > > > > > > > > This is also a critical concern. When executing DDL,
> > > DQL,
> > > > or
> > > > > > DML
> > > > > > > > > > > > statements that reference a specific Catalog, the
> > Catalog
> > > > > > > instance
> > > > > > > > > > > > will be initialized if it has not been initialized
> yet.
> > > If
> > > > > the
> > > > > > > > > > > > initialization process fails, these statements will
> not
> > > be
> > > > > > > executed
> > > > > > > > > > > > successfully.
> > > > > > > > > > > >
> > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs and
> > CatalogStore
> > > > > > > > catalogstore
> > > > > > > > > > in
> > > > > > > > > > > > the CatalogManager all have the same catalog name,
> but
> > > > > > > correponding
> > > > > > > > > to
> > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > >
> > > > > > > > > > > > This issue can be resolved by referring to the
> previous
> > > > > > > responses.
> > > > > > > > > The
> > > > > > > > > > > > key principle is that the Catalog that has been most
> > > > recently
> > > > > > > used
> > > > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for driving this Flip, I do believe this
> Flip
> > > > could
> > > > > be
> > > > > > > > > helpful
> > > > > > > > > > > > for users. The firm I work for also manages a lot of
> > > > > catalogs,
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > submission of tasks becomes slow because of loading a
> > > > number
> > > > > of
> > > > > > > > > > catalogs.
> > > > > > > > > > > > We obtain the catalogs in the user's Flink SQL
> through
> > > > > regular
> > > > > > > > > > expression
> > > > > > > > > > > > to avoid loading all catalogs to improve the speed of
> > > task
> > > > > > > > > submission.
> > > > > > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. When creating a catalog with CREATE CATALOG, the
> > > > > > > asynchronous
> > > > > > > > > > > > registration method is used by default. What if
> > > > asynchronous
> > > > > > > > > > registration
> > > > > > > > > > > > failed? And how to implement asynchronous
> registration?
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. I also have the same question with Shammon’s
> first
> > > > > > comment.
> > > > > > > > The
> > > > > > > > > > > > Map<String, Catalog> catalogs and CatalogStore
> > > catalogstore
> > > > > in
> > > > > > > the
> > > > > > > > > > > > CatalogManager all have the same catalog name, but
> > > > > correponding
> > > > > > > to
> > > > > > > > > > > > different catalog instances. For example: catalog1->
> > > > > > jdbcCatalog
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the
> > > catalogstore.
> > > > > > Would
> > > > > > > > this
> > > > > > > > > > > case
> > > > > > > > > > > > happen? Maybe we should define `catalogs` and
> > > > `catalogstore`
> > > > > > more
> > > > > > > > > > > clearly.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> ruanhang1993@gmail.com
> > >
> > > > 写道：
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Jing,
Thanks for your reply.
> There might be more such issues. I would suggest you completely walk
through the FLIP again and fix those issues
I am very sorry for my carelessness and at the same time, I greatly
appreciate your careful review. I have thoroughly checked the entire FLIP
and made corrections to these issues.
> If I am not mistaken, with the current FLIP design, CatalogManager
could work without Optional CatalogStore being configured.
Yes, in the original design, CatalogStore was not necessary because
CatalogManager used Map<String, Catalog> catalogs to store catalog
instances.
However, this caused inconsistency issues. Therefore, I modified this part
of the design and removed Map<String, Catalog> catalogs from CatalogManager.
At the same time, InMemoryCatalog will serve as the default CatalogStore
to save catalogs in memory and replace the functionality of
Map<String,Catalog>catalogs.
The previous plan that kept Map<String,Catalog>catalogs has been moved to
Rejected Alternatives.
Best,
Feng
On Sun, Apr 30, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid> wrote:
> Hi Feng,
>
> There are still many places contain inconsistent content, e.g.
>
> 1. "Asynchronous registration" is still used.
> 2. The java comment of the method registerCatalog(String catalogName,
> Catalog catalog) in CatalogManager does not tell what the method is doing.
>
>
> There might be more such issues. I would suggest you completely walk
> through the FLIP again and fix those issues.
>
> About the inMemoryCatalogStore, do you mean that you will build the cache
> functionality in the CatalogStore? This is a very different design concept
> from what the current FLIP described. If I am not mistaken, with the
> current FLIP design, CatalogManager could work without Optional
> CatalogStore being configured. That is the reason why I mentioned in the
> last email that the example code wrt the Optional CatalogStore is not
> correct.
>
> Best regards,
> Jing
>
> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <ji...@gmail.com> wrote:
>
> > Hi Jing
> >
> >
> > > There are still some NIT issues in the FLIP
> >
> > Thank you very much for the careful review. I have already made the
> > relevant changes.
> >
> >
> > > Speaking of the conflict issues in the multi-instance scenarios, I am
> > not
> > sure if this is the intended behaviour
> >
> > Currently, there are conflicts in multiple scenarios with the current
> > design. I am thinking whether we should remove 'Map<String, Catalog>' and
> > make Cache the default behavior of InMemoryCatalogStore. This way, users
> > can implement their own CatalogStore to achieve multi-instance
> > non-conflicting scenarios. What do you think?
> >
> >
> >
> > Best,
> > Feng
> >
> > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > Hi Feng,
> > >
> > > Thanks for working on the FLIP. There are still some NIT issues in the
> > FLIP
> > > like:
> > >
> > > 1. Optional<CatalogStore> catalogStore has been used as CatalogStore
> > > instead of Optional in the code example. It should be fine to use it as
> > > pseudo code for now and update it after you submit the PR.
> > > 2. addCatalog(...) is still used somewhere in the rejected section
> which
> > > should be persistContext(...) to keep it consistent.
> > >
> > > Speaking of the conflict issues in the multi-instance scenarios, I am
> not
> > > sure if this is the intended behaviour. If Map<String, Catalog>
> catalogs
> > is
> > > used as a cache, it should be invalid, once the related catalog has
> been
> > > removed from the CatalogStore by another instance. Did I miss
> something?
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <ji...@gmail.com>
> wrote:
> > >
> > > > Hi Jing,Shammon
> > > > Thanks for your reply.
> > > >
> > > > @Jing
> > > >
> > > > > How about persistCatalog()?
> > > > I think this is a good function name, I have updated it in the
> > > > documentation.
> > > >
> > > > > Some common cache features should be implemented
> > > > Thank you for the suggestion. If alternative 1 turns out to be more
> > > > appropriate later, I will improve this part of the content.
> > > >
> > > > > As the above discussion moves forward, the option 2 solution looks
> > more
> > > > like a replacement of option 1
> > > > Yes, after discussing with Shammon offline, we think that solution 2
> > > might
> > > > be more suitable and also avoid any inconsistency issues.
> > > >
> > > > > There are some inconsistent descriptions in the content. Would you
> > > like
> > > > to clean them up?
> > > > I will do my best to improve the document and appreciate your
> > > suggestions.
> > > >
> > > >
> > > >
> > > > @Shammon
> > > > > can you put the unselected option in `Rejected Alternatives`
> > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > >
> > > >
> > > >
> > > > Best
> > > > Feng
> > > >
> > > >
> > > >
> > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com>
> wrote:
> > > >
> > > > > Hi Feng
> > > > >
> > > > > Thanks for your update.
> > > > >
> > > > > I found there are two options in `Proposed Changes`, can you put
> the
> > > > > unselected option in `Rejected Alternatives`? I think this may help
> > us
> > > > > better understand your proposal
> > > > >
> > > > >
> > > > > Best,
> > > > > Shammon FY
> > > > >
> > > > >
> > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge <jing@ververica.com.invalid
> >
> > > > > wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > Thanks for raising this FLIP. I am still confused after
> completely
> > > > > reading
> > > > > > the thread with following questions:
> > > > > >
> > > > > > 1. Naming confusion - registerCatalog() and addCatalog() have no
> > big
> > > > > > difference based on their names. One of them is responsible for
> > data
> > > > > > persistence. How about persistCatalog()?
> > > > > > 2. As you mentioned that Map<String, Catalog> catalogs is used
> as a
> > > > cache
> > > > > > and catalogStore is used for data persistence. I would suggest
> > > > describing
> > > > > > their purpose conceptually and clearly in the FLIP. Some common
> > cache
> > > > > > features should be implemented, i.e. data in the cache and the
> > store
> > > > > should
> > > > > > be consistent. Same Catalog instance should be found in the store
> > and
> > > > in
> > > > > > the cache(either it has been initialized or it will be lazy
> > > > initialized)
> > > > > > for the same catalog name. The consistency will be taken care of
> > > while
> > > > > > updating the catalog.
> > > > > > 3. As the above discussion moves forward, the option 2 solution
> > looks
> > > > > more
> > > > > > like a replacement of option 1, because, afaiu, issues mentioned
> > > > > > previously with option 1 are not solved yet. Do you still want to
> > > > propose
> > > > > > both options and ask for suggestions for both of them?
> > > > > > 4. After you updated the FLIP, there are some inconsistent
> > > descriptions
> > > > > in
> > > > > > the content. Would you like to clean them up? Thanks!
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > >
> > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > hi Shammon
> > > > > > >
> > > > > > > Thank you for your response, and I completely agree with your
> > point
> > > > of
> > > > > > > view.
> > > > > > > Initially, I may have over complicated the whole issue. First
> and
> > > > > > foremost,
> > > > > > > we need to consider the persistence of the Catalog's
> > Configuration.
> > > > > > > If we only need to provide persistence for Catalog
> Configuration,
> > > we
> > > > > can
> > > > > > > add a toConfiguration method to the Catalog interface.
> > > > > > > This method can convert a Catalog instance to a Map<String,
> > String>
> > > > > > > properties, and the default implementation will throw an
> > exception.
> > > > > > >
> > > > > > > public interface Catalog {
> > > > > > > /**
> > > > > > > * Returns a map containing the properties of the catalog
> > object.
> > > > > > > *
> > > > > > > * @return a map containing the properties of the catalog
> > object
> > > > > > > * @throws UnsupportedOperationException if the implementing
> > > class
> > > > > does
> > > > > > > not override
> > > > > > > * the default implementation of this method
> > > > > > > */
> > > > > > > default Map<String, String> toProperties() {
> > > > > > > throw new UnsupportedOperationException("Please implement
> > > > > > toProperties
> > > > > > > for this catalog");
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > > The specific process is as follows:
> > > > > > >
> > > > > > > 1. If the user has configured a CatalogStore, the
> > > toConfiguration()
> > > > > > method
> > > > > > > will be called when registering a Catalog instance with
> > > > > > > registerCatalog(String catalogName, Catalog catalog). The
> Catalog
> > > > > > instance
> > > > > > > will be converted to a Map<String, String> properties and saved
> > to
> > > > the
> > > > > > > CatalogStore. If some Catalog instances do not implement this
> > > method,
> > > > > an
> > > > > > > exception will be thrown.
> > > > > > >
> > > > > > > 2. If the user does not use a CatalogStore, the
> > toConfiguration()
> > > > > method
> > > > > > > will not be called, ensuring consistency with the original
> > process.
> > > > > > >
> > > > > > > 3. Saving both the Map<String, Catalog> catalogs and the
> > > CatalogStore
> > > > > at
> > > > > > > the same time can also avoid conflicts
> > > > > > >
> > > > > > >
> > > > > > > For lazy initialization:
> > > > > > > we can start from the Catalog itself and provide a dedicated
> > > Catalog
> > > > > > > implementation for lazy loading, such as ConfigurationCatalog
> > > > > > >
> > > > > > > public class ConfigurationCatalog implements Catalog {
> > > > > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > > I have added this design to the FLIP.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <zj...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng
> > > > > > > >
> > > > > > > > Thanks for your answer.
> > > > > > > >
> > > > > > > > I have no questions about the functionality of
> `CatalogStore`,
> > > but
> > > > > the
> > > > > > > > different operations of multiple `registerCatalog` or
> > > > `storeCatalog`
> > > > > in
> > > > > > > > `CatalogManager`. The implementation in Trino is also the
> same,
> > > the
> > > > > > > > `CatalogManager` in trino only has one `createCatalog`, which
> > > will
> > > > > save
> > > > > > > the
> > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > >
> > > > > > > > I think we don't need to add another `registerCatalog` or
> > > > > `addCatalog`
> > > > > > to
> > > > > > > > save a catalog in `Map<String, Catalog> catalogs` or
> > > `CatalogStore
> > > > > > > > catalogStore`. As you mentioned before, the `Map<String,
> > > Catalog>
> > > > > > > > catalogs` is a cache in `CatalogManager`. How about saving
> the
> > > > > catalog
> > > > > > in
> > > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore
> catalogStore`
> > > > > > together
> > > > > > > > when it is registered or added in `CatalogManager`?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shammon FY
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> jinfeng1995@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thank you for your reply. I am very sorry for the
> > > > misunderstanding
> > > > > > > caused
> > > > > > > > > by my deviation from the original discussion.
> > > > > > > > >
> > > > > > > > > @Shammon
> > > > > > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > > > > > Yes, there was indeed such a discussion before. However,
> in
> > > > > > designing
> > > > > > > > the
> > > > > > > > > whole solution, I found that the logic of CatalogManager
> > itself
> > > > > > doesn't
> > > > > > > > > need to change much. *We cannot only persist Catalog
> > instances
> > > > > > > > themselves*,
> > > > > > > > > so exposing only registerCatalog(String catalogName,
> Catalog
> > > > > catalog)
> > > > > > > > might
> > > > > > > > > not be enough to save Catalogs, because in the end we still
> > > need
> > > > to
> > > > > > > save
> > > > > > > > > the configurations corresponding to the Catalog instances.
> > > > > > Therefore,
> > > > > > > I
> > > > > > > > > decided to introduce the CatalogStore interface for
> > > configuration
> > > > > > > > > persistence. Regarding this part, I also took inspiration
> > from
> > > > > > Trino's
> > > > > > > > > implementation[1].
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > @yuxia
> > > > > > > > >
> > > > > > > > > > 1: The mechanism of handling catalog with the same name
> > > looks a
> > > > > > > little
> > > > > > > > of
> > > > > > > > > complex to me.
> > > > > > > > > Thank you for the suggestion. I will provide a detailed
> > > > description
> > > > > > and
> > > > > > > > > code examples for this part, and add it to the FLIP.
> > > > > > > > >
> > > > > > > > > > 2: TBH, the method name `addCatalog` still looks confused
> > to
> > > > me.
> > > > > > > IIUC,
> > > > > > > > > this method is for storing catalog to CatalogStore, how
> about
> > > > > > renaming
> > > > > > > it
> > > > > > > > > to `storeCatalog`? It's very personal opinion, you can
> decide
> > > to
> > > > > take
> > > > > > > it
> > > > > > > > or
> > > > > > > > > not by your self.
> > > > > > > > > StoreCatalog looks more intuitive to me. I don't see any
> > > problem
> > > > > with
> > > > > > > it.
> > > > > > > > >
> > > > > > > > > > 3.For CREATE CATALOG statement, which method will be
> > called?
> > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether
> user
> > > can
> > > > > > add a
> > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > For CREATE CATALOG, my original design was to add it
> directly
> > > to
> > > > > the
> > > > > > > > > CatalogStore, but this would disrupt the existing logic.
> > > > > Therefore, I
> > > > > > > > think
> > > > > > > > > we can do both: save the configuration to the CatalogStore
> > and
> > > > > > > initialize
> > > > > > > > > the Catalog instance at the same time
> > > > > > > > >
> > > > > > > > > > 3. Is it really neccessary to provide a default
> > implmentation
> > > > for
> > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > I think it is necessary, otherwise we would need to
> introduce
> > > an
> > > > > > > > additional
> > > > > > > > > Map to store the configuration for lazy loading.
> > > > > > > > >
> > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > I don't think registerCatalog(String catalogName, Catalog
> > > > catalog)
> > > > > > can
> > > > > > > be
> > > > > > > > > made into an asynchronous interface because Catalog is
> > already
> > > an
> > > > > > > > instance.
> > > > > > > > >
> > > > > > > > > > It looks more like lazy initialization for catalog than
> > > > > > asynchronous
> > > > > > > > > registration, right?
> > > > > > > > > Indeed, my description was inaccurate. It should be lazy
> > > > > registration
> > > > > > > > > instead of asynchronous registration. I have already
> updated
> > > the
> > > > > > title
> > > > > > > of
> > > > > > > > > the FLIP.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1].
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> zjureel@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng
> > > > > > > > > >
> > > > > > > > > > I think if there is a `registerCatalog` method in
> > > > > `CatalogManager`,
> > > > > > > it
> > > > > > > > > will
> > > > > > > > > > confuse users whether a method named `addCatalog` or
> > > > > `storeCatalog`
> > > > > > > is
> > > > > > > > > > added.
> > > > > > > > > >
> > > > > > > > > > And as you mentioned, the memory catalog is a `cache`, I
> > > think
> > > > > the
> > > > > > > > > concept
> > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > >
> > > > > > > > > > I found there is a pre-discussion [1] for this FLIP.
> Please
> > > > > correct
> > > > > > > me
> > > > > > > > if
> > > > > > > > > > I'm wrong, IIUC, the conclusion of that discussion is to
> > use
> > > > > > > > > > `CatalogManager` as an interface and implement it for
> > > different
> > > > > > > stores
> > > > > > > > > such
> > > > > > > > > > as memory, file and external system.
> > > > > > > > > >
> > > > > > > > > > I think there is a gap between the current FLIP design
> and
> > > that
> > > > > > > > > conclusion.
> > > > > > > > > > What about the proposal of the discussion in thread [1] ?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shammon FY
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > I have few comments:
> > > > > > > > > > > 1: The mechanism of handling catalog with the same name
> > > > looks a
> > > > > > > > little
> > > > > > > > > of
> > > > > > > > > > > complex to me. I think it'll be better to explain it in
> > the
> > > > > java
> > > > > > > doc
> > > > > > > > of
> > > > > > > > > > > these methods and give a brief example in this FLIP.
> > > > > > > > > > >
> > > > > > > > > > > 2: TBH, the method name `addCatalog` still looks
> confused
> > > to
> > > > > me.
> > > > > > > > IIUC,
> > > > > > > > > > > this method is for storing catalog to CatalogStore, how
> > > about
> > > > > > > > renaming
> > > > > > > > > it
> > > > > > > > > > > to `storeCatalog`? It's very personal opinion, you can
> > > decide
> > > > > to
> > > > > > > take
> > > > > > > > > it
> > > > > > > > > > or
> > > > > > > > > > > not by your self.
> > > > > > > > > > >
> > > > > > > > > > > 3: For CREATE CATALOG statement, which method will be
> > > called?
> > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering
> whether
> > > user
> > > > > can
> > > > > > > > add a
> > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 3: Is it really neccessary to provide a default
> > > implmentation
> > > > > for
> > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > > >
> > > > > > > > > > > > When creating a catalog with CREATE CATALOG, the
> > > > asynchronous
> > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > If asynchronous registration is the default behavior,
> it
> > > > there
> > > > > > any
> > > > > > > > way
> > > > > > > > > > > that user can switch to synchronous registration just
> > like
> > > > > > before?
> > > > > > > > > > > Will both method `addCatalog` and `registerCatalog` be
> > > > > > asynchronous
> > > > > > > > > > > registration?
> > > > > > > > > > >
> > > > > > > > > > > IIUC, in asynchronous registration, it may well that
> > CREATE
> > > > > > CATALOG
> > > > > > > > > > > executes successfully, but then the following CREATE
> > TABLE
> > > > > > > statement
> > > > > > > > > will
> > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > I think it's a break change which should be highlighted
> > in
> > > > this
> > > > > > > FLIP,
> > > > > > > > > may
> > > > > > > > > > > be in compatibility part.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > BTW, by saying asynchronous registration, I would like
> to
> > > > > expect
> > > > > > > > there
> > > > > > > > > > > will be an executor to open or register catalog in the
> > > > > > background,
> > > > > > > > but
> > > > > > > > > > from
> > > > > > > > > > > your previous comments,
> > > > > > > > > > > "the Catalog instance will be initialized if it has not
> > > been
> > > > > > > > > initialized
> > > > > > > > > > > yet. If the initialization process fails, these
> > statements
> > > > will
> > > > > > not
> > > > > > > > be
> > > > > > > > > > > executed successfully."
> > > > > > > > > > > It looks more like lazy initialization for catalog than
> > > > > > > asynchronous
> > > > > > > > > > > registration, right?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Yuxia
> > > > > > > > > > >
> > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > Catalog
> > > > > > > > > configuration
> > > > > > > > > > > and asynchronous registration
> > > > > > > > > > >
> > > > > > > > > > > Hi everyone, Thank you all for your interest in this
> > > DISCUSS.
> > > > > > > > > > >
> > > > > > > > > > > @Shammon
> > > > > > > > > > > > How to handle a catalog with the same name that
> exists
> > > for
> > > > > both
> > > > > > > of
> > > > > > > > > > them?
> > > > > > > > > > >
> > > > > > > > > > > I believe this is a crucial point. Based on my current
> > > > personal
> > > > > > > > > > > understanding, the Map<String, Catalog> catalogs will
> > serve
> > > > as
> > > > > a
> > > > > > > > cache
> > > > > > > > > > > for instantiated catalogs and have the highest
> priority.
> > > > > > > > > > >
> > > > > > > > > > > There are three methods that can affect the Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs:
> > > > > > > > > > >
> > > > > > > > > > > 1. registerCatalog(String catalogName, Catalog catalog)
> > > > > > > > > > >
> > > > > > > > > > > This method puts the catalog instance into the
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs.
> > > > > > > > > > >
> > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > >
> > > > > > > > > > > This method removes the catalog instance corresponding
> to
> > > > > > > catalogName
> > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > >
> > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > >
> > > > > > > > > > > This method first retrieves the corresponding catalog
> > > > instance
> > > > > > from
> > > > > > > > > > > the Map<String, Catalog> catalogs. If the catalog does
> > not
> > > > > exist,
> > > > > > > it
> > > > > > > > > > > retrieves the corresponding configuration from the
> > > > > CatalogStore,
> > > > > > > > > > > initializes it, and puts the initialized Catalog
> instance
> > > > into
> > > > > > the
> > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > >
> > > > > > > > > > > The following two methods only modify the configuration
> > in
> > > > the
> > > > > > > > > > > CatalogStore:
> > > > > > > > > > >
> > > > > > > > > > > 1. addCatalog(String catalogName, Map<String, String>
> > > > > properties)
> > > > > > > > > > >
> > > > > > > > > > > This method saves the properties to the catalogStore
> and
> > > > checks
> > > > > > > > > > > whether there is a catalogName with the same name.
> > > > > > > > > > >
> > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > This method removes the specified configuration of the
> > > > > specified
> > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > >
> > > > > > > > > > > The following are possible conflict scenarios:
> > > > > > > > > > >
> > > > > > > > > > > 1. When the corresponding catalogName already exists in
> > the
> > > > > > > > > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > > > > > > > > registerCatalog(String catalogName, Catalog catalog)
> > method
> > > > can
> > > > > > > > > > > succeed and be directly saved to the Map<String,
> Catalog>
> > > > > > catalogs.
> > > > > > > > > > >
> > > > > > > > > > > 2. When the corresponding catalogName already exists in
> > > both
> > > > > the
> > > > > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > > > > registerCatalog(String
> > > > > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > > > > >
> > > > > > > > > > > 3. When the corresponding catalogName already exists in
> > the
> > > > > > > > > > > Map<String, Catalog>, the addCatalog(String
> catalogName,
> > > > > > > Map<String,
> > > > > > > > > > > String> properties) method can directly save the
> > properties
> > > > to
> > > > > > the
> > > > > > > > > > > catalogStore, but the getCatalog(String catalogName)
> > method
> > > > > will
> > > > > > > not
> > > > > > > > > > > use the new properties for initialization because the
> > > > > > corresponding
> > > > > > > > > > > catalog instance already exists in catalogs and will be
> > > > > > > prioritized.
> > > > > > > > > > > Therefore, using the unregisterCatalog(String
> > catalogName)
> > > > > method
> > > > > > > to
> > > > > > > > > > > remove the instance corresponding to the original
> > > catalogName
> > > > > is
> > > > > > > > > > > necessary.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > I think it will confuse users that
> > > `registerCatalog(String
> > > > > > > > > > > catalogName,Catalog catalog)` in the `Map<String,
> > Catalog>
> > > > > > > catalogs`
> > > > > > > > > and
> > > > > > > > > > > `registerCatalog(String catalogName, Map<String,
> String>
> > > > > > > properties)
> > > > > > > > > > >
> > > > > > > > > > > This could potentially lead to confusion. I suggest
> > > changing
> > > > > the
> > > > > > > > > > > method name, perhaps to addCatalog(String catalogName,
> > > > > > Map<String,
> > > > > > > > > > > String> properties), as previously mentioned
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > @Hang
> > > > > > > > > > > > add `registerCatalog(String catalogName,Catalog
> > catalog,
> > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > >
> > > > > > > > > > > Since a catalog is already an instance, adding the
> > > "lazyInit"
> > > > > > > > > > > parameter to the registerCatalog(String catalogName,
> > > Catalog
> > > > > > > catalog)
> > > > > > > > > > > method may not necessarily result in lazy
> initialization.
> > > > > > > > > > >
> > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > >
> > > > > > > > > > > I think encryption is necessary, but perhaps the
> > encryption
> > > > > logic
> > > > > > > > > > > should be implemented in the CatalogStore when the data
> > is
> > > > > > actually
> > > > > > > > > > > saved. For instance, the FileCatalogStore could encrypt
> > the
> > > > > data
> > > > > > > when
> > > > > > > > > > > it is saved to a file, while the MemoryCatalogStore
> does
> > > not
> > > > > > > require
> > > > > > > > > > > encryption.
> > > > > > > > > > >
> > > > > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > > > > >
> > > > > > > > > > > I think it is necessary to have a MemoryCatalogStore as
> > the
> > > > > > default
> > > > > > > > > > > implementation, which saves Catalog configurations in
> > > memory.
> > > > > > > > > > > Otherwise, if we want to implement asynchronous loading
> > of
> > > > > > Catalog,
> > > > > > > > we
> > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > @Xianxun
> > > > > > > > > > >
> > > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > > >
> > > > > > > > > > > This is also a critical concern. When executing DDL,
> > DQL,
> > > or
> > > > > DML
> > > > > > > > > > > statements that reference a specific Catalog, the
> Catalog
> > > > > > instance
> > > > > > > > > > > will be initialized if it has not been initialized yet.
> > If
> > > > the
> > > > > > > > > > > initialization process fails, these statements will not
> > be
> > > > > > executed
> > > > > > > > > > > successfully.
> > > > > > > > > > >
> > > > > > > > > > > > nt. The Map<String, Catalog> catalogs and
> CatalogStore
> > > > > > > catalogstore
> > > > > > > > > in
> > > > > > > > > > > the CatalogManager all have the same catalog name, but
> > > > > > correponding
> > > > > > > > to
> > > > > > > > > > > different catalog instances.
> > > > > > > > > > >
> > > > > > > > > > > This issue can be resolved by referring to the previous
> > > > > > responses.
> > > > > > > > The
> > > > > > > > > > > key principle is that the Catalog that has been most
> > > recently
> > > > > > used
> > > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for driving this Flip, I do believe this Flip
> > > could
> > > > be
> > > > > > > > helpful
> > > > > > > > > > > for users. The firm I work for also manages a lot of
> > > > catalogs,
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > submission of tasks becomes slow because of loading a
> > > number
> > > > of
> > > > > > > > > catalogs.
> > > > > > > > > > > We obtain the catalogs in the user's Flink SQL through
> > > > regular
> > > > > > > > > expression
> > > > > > > > > > > to avoid loading all catalogs to improve the speed of
> > task
> > > > > > > > submission.
> > > > > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. When creating a catalog with CREATE CATALOG, the
> > > > > > asynchronous
> > > > > > > > > > > registration method is used by default. What if
> > > asynchronous
> > > > > > > > > registration
> > > > > > > > > > > failed? And how to implement asynchronous registration?
> > > > > > > > > > > >
> > > > > > > > > > > > 2. I also have the same question with Shammon’s first
> > > > > comment.
> > > > > > > The
> > > > > > > > > > > Map<String, Catalog> catalogs and CatalogStore
> > catalogstore
> > > > in
> > > > > > the
> > > > > > > > > > > CatalogManager all have the same catalog name, but
> > > > correponding
> > > > > > to
> > > > > > > > > > > different catalog instances. For example: catalog1->
> > > > > jdbcCatalog
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the
> > catalogstore.
> > > > > Would
> > > > > > > this
> > > > > > > > > > case
> > > > > > > > > > > happen? Maybe we should define `catalogs` and
> > > `catalogstore`
> > > > > more
> > > > > > > > > > clearly.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Xianxun
> > > > > > > > > > > >
> > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <ruanhang1993@gmail.com
> >
> > > 写道：
> > > > > > > > > > > > >
> > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Feng,
There are still many places contain inconsistent content, e.g.
1. "Asynchronous registration" is still used.
2. The java comment of the method registerCatalog(String catalogName,
Catalog catalog) in CatalogManager does not tell what the method is doing.
There might be more such issues. I would suggest you completely walk
through the FLIP again and fix those issues.
About the inMemoryCatalogStore, do you mean that you will build the cache
functionality in the CatalogStore? This is a very different design concept
from what the current FLIP described. If I am not mistaken, with the
current FLIP design, CatalogManager could work without Optional
CatalogStore being configured. That is the reason why I mentioned in the
last email that the example code wrt the Optional CatalogStore is not
correct.
Best regards,
Jing
On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <ji...@gmail.com> wrote:
> Hi Jing
>
>
> > There are still some NIT issues in the FLIP
>
> Thank you very much for the careful review. I have already made the
> relevant changes.
>
>
> > Speaking of the conflict issues in the multi-instance scenarios, I am
> not
> sure if this is the intended behaviour
>
> Currently, there are conflicts in multiple scenarios with the current
> design. I am thinking whether we should remove 'Map<String, Catalog>' and
> make Cache the default behavior of InMemoryCatalogStore. This way, users
> can implement their own CatalogStore to achieve multi-instance
> non-conflicting scenarios. What do you think?
>
>
>
> Best,
> Feng
>
> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi Feng,
> >
> > Thanks for working on the FLIP. There are still some NIT issues in the
> FLIP
> > like:
> >
> > 1. Optional<CatalogStore> catalogStore has been used as CatalogStore
> > instead of Optional in the code example. It should be fine to use it as
> > pseudo code for now and update it after you submit the PR.
> > 2. addCatalog(...) is still used somewhere in the rejected section which
> > should be persistContext(...) to keep it consistent.
> >
> > Speaking of the conflict issues in the multi-instance scenarios, I am not
> > sure if this is the intended behaviour. If Map<String, Catalog> catalogs
> is
> > used as a cache, it should be invalid, once the related catalog has been
> > removed from the CatalogStore by another instance. Did I miss something?
> >
> > Best regards,
> > Jing
> >
> > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <ji...@gmail.com> wrote:
> >
> > > Hi Jing,Shammon
> > > Thanks for your reply.
> > >
> > > @Jing
> > >
> > > > How about persistCatalog()?
> > > I think this is a good function name, I have updated it in the
> > > documentation.
> > >
> > > > Some common cache features should be implemented
> > > Thank you for the suggestion. If alternative 1 turns out to be more
> > > appropriate later, I will improve this part of the content.
> > >
> > > > As the above discussion moves forward, the option 2 solution looks
> more
> > > like a replacement of option 1
> > > Yes, after discussing with Shammon offline, we think that solution 2
> > might
> > > be more suitable and also avoid any inconsistency issues.
> > >
> > > > There are some inconsistent descriptions in the content. Would you
> > like
> > > to clean them up?
> > > I will do my best to improve the document and appreciate your
> > suggestions.
> > >
> > >
> > >
> > > @Shammon
> > > > can you put the unselected option in `Rejected Alternatives`
> > > Sure, I have moved it to the `Rejected Alternatives`.
> > >
> > >
> > >
> > > Best
> > > Feng
> > >
> > >
> > >
> > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com> wrote:
> > >
> > > > Hi Feng
> > > >
> > > > Thanks for your update.
> > > >
> > > > I found there are two options in `Proposed Changes`, can you put the
> > > > unselected option in `Rejected Alternatives`? I think this may help
> us
> > > > better understand your proposal
> > > >
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > >
> > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge <ji...@ververica.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi Feng,
> > > > >
> > > > > Thanks for raising this FLIP. I am still confused after completely
> > > > reading
> > > > > the thread with following questions:
> > > > >
> > > > > 1. Naming confusion - registerCatalog() and addCatalog() have no
> big
> > > > > difference based on their names. One of them is responsible for
> data
> > > > > persistence. How about persistCatalog()?
> > > > > 2. As you mentioned that Map<String, Catalog> catalogs is used as a
> > > cache
> > > > > and catalogStore is used for data persistence. I would suggest
> > > describing
> > > > > their purpose conceptually and clearly in the FLIP. Some common
> cache
> > > > > features should be implemented, i.e. data in the cache and the
> store
> > > > should
> > > > > be consistent. Same Catalog instance should be found in the store
> and
> > > in
> > > > > the cache(either it has been initialized or it will be lazy
> > > initialized)
> > > > > for the same catalog name. The consistency will be taken care of
> > while
> > > > > updating the catalog.
> > > > > 3. As the above discussion moves forward, the option 2 solution
> looks
> > > > more
> > > > > like a replacement of option 1, because, afaiu, issues mentioned
> > > > > previously with option 1 are not solved yet. Do you still want to
> > > propose
> > > > > both options and ask for suggestions for both of them?
> > > > > 4. After you updated the FLIP, there are some inconsistent
> > descriptions
> > > > in
> > > > > the content. Would you like to clean them up? Thanks!
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > >
> > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <ji...@gmail.com>
> > wrote:
> > > > >
> > > > > > hi Shammon
> > > > > >
> > > > > > Thank you for your response, and I completely agree with your
> point
> > > of
> > > > > > view.
> > > > > > Initially, I may have over complicated the whole issue. First and
> > > > > foremost,
> > > > > > we need to consider the persistence of the Catalog's
> Configuration.
> > > > > > If we only need to provide persistence for Catalog Configuration,
> > we
> > > > can
> > > > > > add a toConfiguration method to the Catalog interface.
> > > > > > This method can convert a Catalog instance to a Map<String,
> String>
> > > > > > properties, and the default implementation will throw an
> exception.
> > > > > >
> > > > > > public interface Catalog {
> > > > > > /**
> > > > > > * Returns a map containing the properties of the catalog
> object.
> > > > > > *
> > > > > > * @return a map containing the properties of the catalog
> object
> > > > > > * @throws UnsupportedOperationException if the implementing
> > class
> > > > does
> > > > > > not override
> > > > > > * the default implementation of this method
> > > > > > */
> > > > > > default Map<String, String> toProperties() {
> > > > > > throw new UnsupportedOperationException("Please implement
> > > > > toProperties
> > > > > > for this catalog");
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > The specific process is as follows:
> > > > > >
> > > > > > 1. If the user has configured a CatalogStore, the
> > toConfiguration()
> > > > > method
> > > > > > will be called when registering a Catalog instance with
> > > > > > registerCatalog(String catalogName, Catalog catalog). The Catalog
> > > > > instance
> > > > > > will be converted to a Map<String, String> properties and saved
> to
> > > the
> > > > > > CatalogStore. If some Catalog instances do not implement this
> > method,
> > > > an
> > > > > > exception will be thrown.
> > > > > >
> > > > > > 2. If the user does not use a CatalogStore, the
> toConfiguration()
> > > > method
> > > > > > will not be called, ensuring consistency with the original
> process.
> > > > > >
> > > > > > 3. Saving both the Map<String, Catalog> catalogs and the
> > CatalogStore
> > > > at
> > > > > > the same time can also avoid conflicts
> > > > > >
> > > > > >
> > > > > > For lazy initialization:
> > > > > > we can start from the Catalog itself and provide a dedicated
> > Catalog
> > > > > > implementation for lazy loading, such as ConfigurationCatalog
> > > > > >
> > > > > > public class ConfigurationCatalog implements Catalog {
> > > > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > I have added this design to the FLIP.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <zj...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi Feng
> > > > > > >
> > > > > > > Thanks for your answer.
> > > > > > >
> > > > > > > I have no questions about the functionality of `CatalogStore`,
> > but
> > > > the
> > > > > > > different operations of multiple `registerCatalog` or
> > > `storeCatalog`
> > > > in
> > > > > > > `CatalogManager`. The implementation in Trino is also the same,
> > the
> > > > > > > `CatalogManager` in trino only has one `createCatalog`, which
> > will
> > > > save
> > > > > > the
> > > > > > > catalog to memory and `CatalogStore`.
> > > > > > >
> > > > > > > I think we don't need to add another `registerCatalog` or
> > > > `addCatalog`
> > > > > to
> > > > > > > save a catalog in `Map<String, Catalog> catalogs` or
> > `CatalogStore
> > > > > > > catalogStore`. As you mentioned before, the `Map<String,
> > Catalog>
> > > > > > > catalogs` is a cache in `CatalogManager`. How about saving the
> > > > catalog
> > > > > in
> > > > > > > `Map<String, Catalog> catalogs` and `CatalogStore catalogStore`
> > > > > together
> > > > > > > when it is registered or added in `CatalogManager`?
> > > > > > >
> > > > > > > Best,
> > > > > > > Shammon FY
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <jinfeng1995@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Thank you for your reply. I am very sorry for the
> > > misunderstanding
> > > > > > caused
> > > > > > > > by my deviation from the original discussion.
> > > > > > > >
> > > > > > > > @Shammon
> > > > > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > > > > Yes, there was indeed such a discussion before. However, in
> > > > > designing
> > > > > > > the
> > > > > > > > whole solution, I found that the logic of CatalogManager
> itself
> > > > > doesn't
> > > > > > > > need to change much. *We cannot only persist Catalog
> instances
> > > > > > > themselves*,
> > > > > > > > so exposing only registerCatalog(String catalogName, Catalog
> > > > catalog)
> > > > > > > might
> > > > > > > > not be enough to save Catalogs, because in the end we still
> > need
> > > to
> > > > > > save
> > > > > > > > the configurations corresponding to the Catalog instances.
> > > > > Therefore,
> > > > > > I
> > > > > > > > decided to introduce the CatalogStore interface for
> > configuration
> > > > > > > > persistence. Regarding this part, I also took inspiration
> from
> > > > > Trino's
> > > > > > > > implementation[1].
> > > > > > > >
> > > > > > > >
> > > > > > > > @yuxia
> > > > > > > >
> > > > > > > > > 1: The mechanism of handling catalog with the same name
> > looks a
> > > > > > little
> > > > > > > of
> > > > > > > > complex to me.
> > > > > > > > Thank you for the suggestion. I will provide a detailed
> > > description
> > > > > and
> > > > > > > > code examples for this part, and add it to the FLIP.
> > > > > > > >
> > > > > > > > > 2: TBH, the method name `addCatalog` still looks confused
> to
> > > me.
> > > > > > IIUC,
> > > > > > > > this method is for storing catalog to CatalogStore, how about
> > > > > renaming
> > > > > > it
> > > > > > > > to `storeCatalog`? It's very personal opinion, you can decide
> > to
> > > > take
> > > > > > it
> > > > > > > or
> > > > > > > > not by your self.
> > > > > > > > StoreCatalog looks more intuitive to me. I don't see any
> > problem
> > > > with
> > > > > > it.
> > > > > > > >
> > > > > > > > > 3.For CREATE CATALOG statement, which method will be
> called?
> > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether user
> > can
> > > > > add a
> > > > > > > > catalog to store with SQL stement.
> > > > > > > > For CREATE CATALOG, my original design was to add it directly
> > to
> > > > the
> > > > > > > > CatalogStore, but this would disrupt the existing logic.
> > > > Therefore, I
> > > > > > > think
> > > > > > > > we can do both: save the configuration to the CatalogStore
> and
> > > > > > initialize
> > > > > > > > the Catalog instance at the same time
> > > > > > > >
> > > > > > > > > 3. Is it really neccessary to provide a default
> implmentation
> > > for
> > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > I think it is necessary, otherwise we would need to introduce
> > an
> > > > > > > additional
> > > > > > > > Map to store the configuration for lazy loading.
> > > > > > > >
> > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > I don't think registerCatalog(String catalogName, Catalog
> > > catalog)
> > > > > can
> > > > > > be
> > > > > > > > made into an asynchronous interface because Catalog is
> already
> > an
> > > > > > > instance.
> > > > > > > >
> > > > > > > > > It looks more like lazy initialization for catalog than
> > > > > asynchronous
> > > > > > > > registration, right?
> > > > > > > > Indeed, my description was inaccurate. It should be lazy
> > > > registration
> > > > > > > > instead of asynchronous registration. I have already updated
> > the
> > > > > title
> > > > > > of
> > > > > > > > the FLIP.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > [1].
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zjureel@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng
> > > > > > > > >
> > > > > > > > > I think if there is a `registerCatalog` method in
> > > > `CatalogManager`,
> > > > > > it
> > > > > > > > will
> > > > > > > > > confuse users whether a method named `addCatalog` or
> > > > `storeCatalog`
> > > > > > is
> > > > > > > > > added.
> > > > > > > > >
> > > > > > > > > And as you mentioned, the memory catalog is a `cache`, I
> > think
> > > > the
> > > > > > > > concept
> > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > >
> > > > > > > > > I found there is a pre-discussion [1] for this FLIP. Please
> > > > correct
> > > > > > me
> > > > > > > if
> > > > > > > > > I'm wrong, IIUC, the conclusion of that discussion is to
> use
> > > > > > > > > `CatalogManager` as an interface and implement it for
> > different
> > > > > > stores
> > > > > > > > such
> > > > > > > > > as memory, file and external system.
> > > > > > > > >
> > > > > > > > > I think there is a gap between the current FLIP design and
> > that
> > > > > > > > conclusion.
> > > > > > > > > What about the proposal of the discussion in thread [1] ?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]
> > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shammon FY
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > I have few comments:
> > > > > > > > > > 1: The mechanism of handling catalog with the same name
> > > looks a
> > > > > > > little
> > > > > > > > of
> > > > > > > > > > complex to me. I think it'll be better to explain it in
> the
> > > > java
> > > > > > doc
> > > > > > > of
> > > > > > > > > > these methods and give a brief example in this FLIP.
> > > > > > > > > >
> > > > > > > > > > 2: TBH, the method name `addCatalog` still looks confused
> > to
> > > > me.
> > > > > > > IIUC,
> > > > > > > > > > this method is for storing catalog to CatalogStore, how
> > about
> > > > > > > renaming
> > > > > > > > it
> > > > > > > > > > to `storeCatalog`? It's very personal opinion, you can
> > decide
> > > > to
> > > > > > take
> > > > > > > > it
> > > > > > > > > or
> > > > > > > > > > not by your self.
> > > > > > > > > >
> > > > > > > > > > 3: For CREATE CATALOG statement, which method will be
> > called?
> > > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether
> > user
> > > > can
> > > > > > > add a
> > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 3: Is it really neccessary to provide a default
> > implmentation
> > > > for
> > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > > >
> > > > > > > > > > > When creating a catalog with CREATE CATALOG, the
> > > asynchronous
> > > > > > > > > > registration method is used by default.
> > > > > > > > > > If asynchronous registration is the default behavior, it
> > > there
> > > > > any
> > > > > > > way
> > > > > > > > > > that user can switch to synchronous registration just
> like
> > > > > before?
> > > > > > > > > > Will both method `addCatalog` and `registerCatalog` be
> > > > > asynchronous
> > > > > > > > > > registration?
> > > > > > > > > >
> > > > > > > > > > IIUC, in asynchronous registration, it may well that
> CREATE
> > > > > CATALOG
> > > > > > > > > > executes successfully, but then the following CREATE
> TABLE
> > > > > > statement
> > > > > > > > will
> > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > I think it's a break change which should be highlighted
> in
> > > this
> > > > > > FLIP,
> > > > > > > > may
> > > > > > > > > > be in compatibility part.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > BTW, by saying asynchronous registration, I would like to
> > > > expect
> > > > > > > there
> > > > > > > > > > will be an executor to open or register catalog in the
> > > > > background,
> > > > > > > but
> > > > > > > > > from
> > > > > > > > > > your previous comments,
> > > > > > > > > > "the Catalog instance will be initialized if it has not
> > been
> > > > > > > > initialized
> > > > > > > > > > yet. If the initialization process fails, these
> statements
> > > will
> > > > > not
> > > > > > > be
> > > > > > > > > > executed successfully."
> > > > > > > > > > It looks more like lazy initialization for catalog than
> > > > > > asynchronous
> > > > > > > > > > registration, right?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Yuxia
> > > > > > > > > >
> > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> Catalog
> > > > > > > > configuration
> > > > > > > > > > and asynchronous registration
> > > > > > > > > >
> > > > > > > > > > Hi everyone, Thank you all for your interest in this
> > DISCUSS.
> > > > > > > > > >
> > > > > > > > > > @Shammon
> > > > > > > > > > > How to handle a catalog with the same name that exists
> > for
> > > > both
> > > > > > of
> > > > > > > > > them?
> > > > > > > > > >
> > > > > > > > > > I believe this is a crucial point. Based on my current
> > > personal
> > > > > > > > > > understanding, the Map<String, Catalog> catalogs will
> serve
> > > as
> > > > a
> > > > > > > cache
> > > > > > > > > > for instantiated catalogs and have the highest priority.
> > > > > > > > > >
> > > > > > > > > > There are three methods that can affect the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs:
> > > > > > > > > >
> > > > > > > > > > 1. registerCatalog(String catalogName, Catalog catalog)
> > > > > > > > > >
> > > > > > > > > > This method puts the catalog instance into the
> Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs.
> > > > > > > > > >
> > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > >
> > > > > > > > > > This method removes the catalog instance corresponding to
> > > > > > catalogName
> > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > >
> > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > >
> > > > > > > > > > This method first retrieves the corresponding catalog
> > > instance
> > > > > from
> > > > > > > > > > the Map<String, Catalog> catalogs. If the catalog does
> not
> > > > exist,
> > > > > > it
> > > > > > > > > > retrieves the corresponding configuration from the
> > > > CatalogStore,
> > > > > > > > > > initializes it, and puts the initialized Catalog instance
> > > into
> > > > > the
> > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > >
> > > > > > > > > > The following two methods only modify the configuration
> in
> > > the
> > > > > > > > > > CatalogStore:
> > > > > > > > > >
> > > > > > > > > > 1. addCatalog(String catalogName, Map<String, String>
> > > > properties)
> > > > > > > > > >
> > > > > > > > > > This method saves the properties to the catalogStore and
> > > checks
> > > > > > > > > > whether there is a catalogName with the same name.
> > > > > > > > > >
> > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > This method removes the specified configuration of the
> > > > specified
> > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > >
> > > > > > > > > > The following are possible conflict scenarios:
> > > > > > > > > >
> > > > > > > > > > 1. When the corresponding catalogName already exists in
> the
> > > > > > > > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > > > > > > > registerCatalog(String catalogName, Catalog catalog)
> method
> > > can
> > > > > > > > > > succeed and be directly saved to the Map<String, Catalog>
> > > > > catalogs.
> > > > > > > > > >
> > > > > > > > > > 2. When the corresponding catalogName already exists in
> > both
> > > > the
> > > > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > > > registerCatalog(String
> > > > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > > > >
> > > > > > > > > > 3. When the corresponding catalogName already exists in
> the
> > > > > > > > > > Map<String, Catalog>, the addCatalog(String catalogName,
> > > > > > Map<String,
> > > > > > > > > > String> properties) method can directly save the
> properties
> > > to
> > > > > the
> > > > > > > > > > catalogStore, but the getCatalog(String catalogName)
> method
> > > > will
> > > > > > not
> > > > > > > > > > use the new properties for initialization because the
> > > > > corresponding
> > > > > > > > > > catalog instance already exists in catalogs and will be
> > > > > > prioritized.
> > > > > > > > > > Therefore, using the unregisterCatalog(String
> catalogName)
> > > > method
> > > > > > to
> > > > > > > > > > remove the instance corresponding to the original
> > catalogName
> > > > is
> > > > > > > > > > necessary.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > I think it will confuse users that
> > `registerCatalog(String
> > > > > > > > > > catalogName,Catalog catalog)` in the `Map<String,
> Catalog>
> > > > > > catalogs`
> > > > > > > > and
> > > > > > > > > > `registerCatalog(String catalogName, Map<String, String>
> > > > > > properties)
> > > > > > > > > >
> > > > > > > > > > This could potentially lead to confusion. I suggest
> > changing
> > > > the
> > > > > > > > > > method name, perhaps to addCatalog(String catalogName,
> > > > > Map<String,
> > > > > > > > > > String> properties), as previously mentioned
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > @Hang
> > > > > > > > > > > add `registerCatalog(String catalogName,Catalog
> catalog,
> > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > >
> > > > > > > > > > Since a catalog is already an instance, adding the
> > "lazyInit"
> > > > > > > > > > parameter to the registerCatalog(String catalogName,
> > Catalog
> > > > > > catalog)
> > > > > > > > > > method may not necessarily result in lazy initialization.
> > > > > > > > > >
> > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > >
> > > > > > > > > > I think encryption is necessary, but perhaps the
> encryption
> > > > logic
> > > > > > > > > > should be implemented in the CatalogStore when the data
> is
> > > > > actually
> > > > > > > > > > saved. For instance, the FileCatalogStore could encrypt
> the
> > > > data
> > > > > > when
> > > > > > > > > > it is saved to a file, while the MemoryCatalogStore does
> > not
> > > > > > require
> > > > > > > > > > encryption.
> > > > > > > > > >
> > > > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > > > >
> > > > > > > > > > I think it is necessary to have a MemoryCatalogStore as
> the
> > > > > default
> > > > > > > > > > implementation, which saves Catalog configurations in
> > memory.
> > > > > > > > > > Otherwise, if we want to implement asynchronous loading
> of
> > > > > Catalog,
> > > > > > > we
> > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > @Xianxun
> > > > > > > > > >
> > > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > > >
> > > > > > > > > > This is also a critical concern. When executing DDL,
> DQL,
> > or
> > > > DML
> > > > > > > > > > statements that reference a specific Catalog, the Catalog
> > > > > instance
> > > > > > > > > > will be initialized if it has not been initialized yet.
> If
> > > the
> > > > > > > > > > initialization process fails, these statements will not
> be
> > > > > executed
> > > > > > > > > > successfully.
> > > > > > > > > >
> > > > > > > > > > > nt. The Map<String, Catalog> catalogs and CatalogStore
> > > > > > catalogstore
> > > > > > > > in
> > > > > > > > > > the CatalogManager all have the same catalog name, but
> > > > > correponding
> > > > > > > to
> > > > > > > > > > different catalog instances.
> > > > > > > > > >
> > > > > > > > > > This issue can be resolved by referring to the previous
> > > > > responses.
> > > > > > > The
> > > > > > > > > > key principle is that the Catalog that has been most
> > recently
> > > > > used
> > > > > > > > > > will be given priority for subsequent use.
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > > yesorno828423@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for driving this Flip, I do believe this Flip
> > could
> > > be
> > > > > > > helpful
> > > > > > > > > > for users. The firm I work for also manages a lot of
> > > catalogs,
> > > > > and
> > > > > > > the
> > > > > > > > > > submission of tasks becomes slow because of loading a
> > number
> > > of
> > > > > > > > catalogs.
> > > > > > > > > > We obtain the catalogs in the user's Flink SQL through
> > > regular
> > > > > > > > expression
> > > > > > > > > > to avoid loading all catalogs to improve the speed of
> task
> > > > > > > submission.
> > > > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > > > >
> > > > > > > > > > > 1. When creating a catalog with CREATE CATALOG, the
> > > > > asynchronous
> > > > > > > > > > registration method is used by default. What if
> > asynchronous
> > > > > > > > registration
> > > > > > > > > > failed? And how to implement asynchronous registration?
> > > > > > > > > > >
> > > > > > > > > > > 2. I also have the same question with Shammon’s first
> > > > comment.
> > > > > > The
> > > > > > > > > > Map<String, Catalog> catalogs and CatalogStore
> catalogstore
> > > in
> > > > > the
> > > > > > > > > > CatalogManager all have the same catalog name, but
> > > correponding
> > > > > to
> > > > > > > > > > different catalog instances. For example: catalog1->
> > > > jdbcCatalog
> > > > > in
> > > > > > > the
> > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the
> catalogstore.
> > > > Would
> > > > > > this
> > > > > > > > > case
> > > > > > > > > > happen? Maybe we should define `catalogs` and
> > `catalogstore`
> > > > more
> > > > > > > > > clearly.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > >
> > > > > > > > > > > Xianxun
> > > > > > > > > > >
> > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com>
> > 写道：
> > > > > > > > > > > >
> > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Jing
> There are still some NIT issues in the FLIP
Thank you very much for the careful review. I have already made the
relevant changes.
> Speaking of the conflict issues in the multi-instance scenarios, I am
not
sure if this is the intended behaviour
Currently, there are conflicts in multiple scenarios with the current
design. I am thinking whether we should remove 'Map<String, Catalog>' and
make Cache the default behavior of InMemoryCatalogStore. This way, users
can implement their own CatalogStore to achieve multi-instance
non-conflicting scenarios. What do you think?
Best,
Feng
On Thu, Apr 27, 2023 at 9:03 PM Jing Ge <ji...@ververica.com.invalid> wrote:
> Hi Feng,
>
> Thanks for working on the FLIP. There are still some NIT issues in the FLIP
> like:
>
> 1. Optional<CatalogStore> catalogStore has been used as CatalogStore
> instead of Optional in the code example. It should be fine to use it as
> pseudo code for now and update it after you submit the PR.
> 2. addCatalog(...) is still used somewhere in the rejected section which
> should be persistContext(...) to keep it consistent.
>
> Speaking of the conflict issues in the multi-instance scenarios, I am not
> sure if this is the intended behaviour. If Map<String, Catalog> catalogs is
> used as a cache, it should be invalid, once the related catalog has been
> removed from the CatalogStore by another instance. Did I miss something?
>
> Best regards,
> Jing
>
> On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <ji...@gmail.com> wrote:
>
> > Hi Jing,Shammon
> > Thanks for your reply.
> >
> > @Jing
> >
> > > How about persistCatalog()?
> > I think this is a good function name, I have updated it in the
> > documentation.
> >
> > > Some common cache features should be implemented
> > Thank you for the suggestion. If alternative 1 turns out to be more
> > appropriate later, I will improve this part of the content.
> >
> > > As the above discussion moves forward, the option 2 solution looks more
> > like a replacement of option 1
> > Yes, after discussing with Shammon offline, we think that solution 2
> might
> > be more suitable and also avoid any inconsistency issues.
> >
> > > There are some inconsistent descriptions in the content. Would you
> like
> > to clean them up?
> > I will do my best to improve the document and appreciate your
> suggestions.
> >
> >
> >
> > @Shammon
> > > can you put the unselected option in `Rejected Alternatives`
> > Sure, I have moved it to the `Rejected Alternatives`.
> >
> >
> >
> > Best
> > Feng
> >
> >
> >
> > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com> wrote:
> >
> > > Hi Feng
> > >
> > > Thanks for your update.
> > >
> > > I found there are two options in `Proposed Changes`, can you put the
> > > unselected option in `Rejected Alternatives`? I think this may help us
> > > better understand your proposal
> > >
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > >
> > > > Hi Feng,
> > > >
> > > > Thanks for raising this FLIP. I am still confused after completely
> > > reading
> > > > the thread with following questions:
> > > >
> > > > 1. Naming confusion - registerCatalog() and addCatalog() have no big
> > > > difference based on their names. One of them is responsible for data
> > > > persistence. How about persistCatalog()?
> > > > 2. As you mentioned that Map<String, Catalog> catalogs is used as a
> > cache
> > > > and catalogStore is used for data persistence. I would suggest
> > describing
> > > > their purpose conceptually and clearly in the FLIP. Some common cache
> > > > features should be implemented, i.e. data in the cache and the store
> > > should
> > > > be consistent. Same Catalog instance should be found in the store and
> > in
> > > > the cache(either it has been initialized or it will be lazy
> > initialized)
> > > > for the same catalog name. The consistency will be taken care of
> while
> > > > updating the catalog.
> > > > 3. As the above discussion moves forward, the option 2 solution looks
> > > more
> > > > like a replacement of option 1, because, afaiu, issues mentioned
> > > > previously with option 1 are not solved yet. Do you still want to
> > propose
> > > > both options and ask for suggestions for both of them?
> > > > 4. After you updated the FLIP, there are some inconsistent
> descriptions
> > > in
> > > > the content. Would you like to clean them up? Thanks!
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <ji...@gmail.com>
> wrote:
> > > >
> > > > > hi Shammon
> > > > >
> > > > > Thank you for your response, and I completely agree with your point
> > of
> > > > > view.
> > > > > Initially, I may have over complicated the whole issue. First and
> > > > foremost,
> > > > > we need to consider the persistence of the Catalog's Configuration.
> > > > > If we only need to provide persistence for Catalog Configuration,
> we
> > > can
> > > > > add a toConfiguration method to the Catalog interface.
> > > > > This method can convert a Catalog instance to a Map<String, String>
> > > > > properties, and the default implementation will throw an exception.
> > > > >
> > > > > public interface Catalog {
> > > > > /**
> > > > > * Returns a map containing the properties of the catalog object.
> > > > > *
> > > > > * @return a map containing the properties of the catalog object
> > > > > * @throws UnsupportedOperationException if the implementing
> class
> > > does
> > > > > not override
> > > > > * the default implementation of this method
> > > > > */
> > > > > default Map<String, String> toProperties() {
> > > > > throw new UnsupportedOperationException("Please implement
> > > > toProperties
> > > > > for this catalog");
> > > > > }
> > > > > }
> > > > >
> > > > > The specific process is as follows:
> > > > >
> > > > > 1. If the user has configured a CatalogStore, the
> toConfiguration()
> > > > method
> > > > > will be called when registering a Catalog instance with
> > > > > registerCatalog(String catalogName, Catalog catalog). The Catalog
> > > > instance
> > > > > will be converted to a Map<String, String> properties and saved to
> > the
> > > > > CatalogStore. If some Catalog instances do not implement this
> method,
> > > an
> > > > > exception will be thrown.
> > > > >
> > > > > 2. If the user does not use a CatalogStore, the toConfiguration()
> > > method
> > > > > will not be called, ensuring consistency with the original process.
> > > > >
> > > > > 3. Saving both the Map<String, Catalog> catalogs and the
> CatalogStore
> > > at
> > > > > the same time can also avoid conflicts
> > > > >
> > > > >
> > > > > For lazy initialization:
> > > > > we can start from the Catalog itself and provide a dedicated
> Catalog
> > > > > implementation for lazy loading, such as ConfigurationCatalog
> > > > >
> > > > > public class ConfigurationCatalog implements Catalog {
> > > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > > }
> > > > > }
> > > > >
> > > > > I have added this design to the FLIP.
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <zj...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Feng
> > > > > >
> > > > > > Thanks for your answer.
> > > > > >
> > > > > > I have no questions about the functionality of `CatalogStore`,
> but
> > > the
> > > > > > different operations of multiple `registerCatalog` or
> > `storeCatalog`
> > > in
> > > > > > `CatalogManager`. The implementation in Trino is also the same,
> the
> > > > > > `CatalogManager` in trino only has one `createCatalog`, which
> will
> > > save
> > > > > the
> > > > > > catalog to memory and `CatalogStore`.
> > > > > >
> > > > > > I think we don't need to add another `registerCatalog` or
> > > `addCatalog`
> > > > to
> > > > > > save a catalog in `Map<String, Catalog> catalogs` or
> `CatalogStore
> > > > > > catalogStore`. As you mentioned before, the `Map<String,
> Catalog>
> > > > > > catalogs` is a cache in `CatalogManager`. How about saving the
> > > catalog
> > > > in
> > > > > > `Map<String, Catalog> catalogs` and `CatalogStore catalogStore`
> > > > together
> > > > > > when it is registered or added in `CatalogManager`?
> > > > > >
> > > > > > Best,
> > > > > > Shammon FY
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Thank you for your reply. I am very sorry for the
> > misunderstanding
> > > > > caused
> > > > > > > by my deviation from the original discussion.
> > > > > > >
> > > > > > > @Shammon
> > > > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > > > Yes, there was indeed such a discussion before. However, in
> > > > designing
> > > > > > the
> > > > > > > whole solution, I found that the logic of CatalogManager itself
> > > > doesn't
> > > > > > > need to change much. *We cannot only persist Catalog instances
> > > > > > themselves*,
> > > > > > > so exposing only registerCatalog(String catalogName, Catalog
> > > catalog)
> > > > > > might
> > > > > > > not be enough to save Catalogs, because in the end we still
> need
> > to
> > > > > save
> > > > > > > the configurations corresponding to the Catalog instances.
> > > > Therefore,
> > > > > I
> > > > > > > decided to introduce the CatalogStore interface for
> configuration
> > > > > > > persistence. Regarding this part, I also took inspiration from
> > > > Trino's
> > > > > > > implementation[1].
> > > > > > >
> > > > > > >
> > > > > > > @yuxia
> > > > > > >
> > > > > > > > 1: The mechanism of handling catalog with the same name
> looks a
> > > > > little
> > > > > > of
> > > > > > > complex to me.
> > > > > > > Thank you for the suggestion. I will provide a detailed
> > description
> > > > and
> > > > > > > code examples for this part, and add it to the FLIP.
> > > > > > >
> > > > > > > > 2: TBH, the method name `addCatalog` still looks confused to
> > me.
> > > > > IIUC,
> > > > > > > this method is for storing catalog to CatalogStore, how about
> > > > renaming
> > > > > it
> > > > > > > to `storeCatalog`? It's very personal opinion, you can decide
> to
> > > take
> > > > > it
> > > > > > or
> > > > > > > not by your self.
> > > > > > > StoreCatalog looks more intuitive to me. I don't see any
> problem
> > > with
> > > > > it.
> > > > > > >
> > > > > > > > 3.For CREATE CATALOG statement, which method will be called?
> > > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether user
> can
> > > > add a
> > > > > > > catalog to store with SQL stement.
> > > > > > > For CREATE CATALOG, my original design was to add it directly
> to
> > > the
> > > > > > > CatalogStore, but this would disrupt the existing logic.
> > > Therefore, I
> > > > > > think
> > > > > > > we can do both: save the configuration to the CatalogStore and
> > > > > initialize
> > > > > > > the Catalog instance at the same time
> > > > > > >
> > > > > > > > 3. Is it really neccessary to provide a default implmentation
> > for
> > > > > > > interface `CatalogStoreFactory`?
> > > > > > > I think it is necessary, otherwise we would need to introduce
> an
> > > > > > additional
> > > > > > > Map to store the configuration for lazy loading.
> > > > > > >
> > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > I don't think registerCatalog(String catalogName, Catalog
> > catalog)
> > > > can
> > > > > be
> > > > > > > made into an asynchronous interface because Catalog is already
> an
> > > > > > instance.
> > > > > > >
> > > > > > > > It looks more like lazy initialization for catalog than
> > > > asynchronous
> > > > > > > registration, right?
> > > > > > > Indeed, my description was inaccurate. It should be lazy
> > > registration
> > > > > > > instead of asynchronous registration. I have already updated
> the
> > > > title
> > > > > of
> > > > > > > the FLIP.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > [1].
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zj...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Feng
> > > > > > > >
> > > > > > > > I think if there is a `registerCatalog` method in
> > > `CatalogManager`,
> > > > > it
> > > > > > > will
> > > > > > > > confuse users whether a method named `addCatalog` or
> > > `storeCatalog`
> > > > > is
> > > > > > > > added.
> > > > > > > >
> > > > > > > > And as you mentioned, the memory catalog is a `cache`, I
> think
> > > the
> > > > > > > concept
> > > > > > > > of `cache` should not be exposed to users.
> > > > > > > >
> > > > > > > > I found there is a pre-discussion [1] for this FLIP. Please
> > > correct
> > > > > me
> > > > > > if
> > > > > > > > I'm wrong, IIUC, the conclusion of that discussion is to use
> > > > > > > > `CatalogManager` as an interface and implement it for
> different
> > > > > stores
> > > > > > > such
> > > > > > > > as memory, file and external system.
> > > > > > > >
> > > > > > > > I think there is a gap between the current FLIP design and
> that
> > > > > > > conclusion.
> > > > > > > > What about the proposal of the discussion in thread [1] ?
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shammon FY
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > I have few comments:
> > > > > > > > > 1: The mechanism of handling catalog with the same name
> > looks a
> > > > > > little
> > > > > > > of
> > > > > > > > > complex to me. I think it'll be better to explain it in the
> > > java
> > > > > doc
> > > > > > of
> > > > > > > > > these methods and give a brief example in this FLIP.
> > > > > > > > >
> > > > > > > > > 2: TBH, the method name `addCatalog` still looks confused
> to
> > > me.
> > > > > > IIUC,
> > > > > > > > > this method is for storing catalog to CatalogStore, how
> about
> > > > > > renaming
> > > > > > > it
> > > > > > > > > to `storeCatalog`? It's very personal opinion, you can
> decide
> > > to
> > > > > take
> > > > > > > it
> > > > > > > > or
> > > > > > > > > not by your self.
> > > > > > > > >
> > > > > > > > > 3: For CREATE CATALOG statement, which method will be
> called?
> > > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether
> user
> > > can
> > > > > > add a
> > > > > > > > > catalog to store with SQL stement.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 3: Is it really neccessary to provide a default
> implmentation
> > > for
> > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > > >
> > > > > > > > > > When creating a catalog with CREATE CATALOG, the
> > asynchronous
> > > > > > > > > registration method is used by default.
> > > > > > > > > If asynchronous registration is the default behavior, it
> > there
> > > > any
> > > > > > way
> > > > > > > > > that user can switch to synchronous registration just like
> > > > before?
> > > > > > > > > Will both method `addCatalog` and `registerCatalog` be
> > > > asynchronous
> > > > > > > > > registration?
> > > > > > > > >
> > > > > > > > > IIUC, in asynchronous registration, it may well that CREATE
> > > > CATALOG
> > > > > > > > > executes successfully, but then the following CREATE TABLE
> > > > > statement
> > > > > > > will
> > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > I think it's a break change which should be highlighted in
> > this
> > > > > FLIP,
> > > > > > > may
> > > > > > > > > be in compatibility part.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > BTW, by saying asynchronous registration, I would like to
> > > expect
> > > > > > there
> > > > > > > > > will be an executor to open or register catalog in the
> > > > background,
> > > > > > but
> > > > > > > > from
> > > > > > > > > your previous comments,
> > > > > > > > > "the Catalog instance will be initialized if it has not
> been
> > > > > > > initialized
> > > > > > > > > yet. If the initialization process fails, these statements
> > will
> > > > not
> > > > > > be
> > > > > > > > > executed successfully."
> > > > > > > > > It looks more like lazy initialization for catalog than
> > > > > asynchronous
> > > > > > > > > registration, right?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Yuxia
> > > > > > > > >
> > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > > > configuration
> > > > > > > > > and asynchronous registration
> > > > > > > > >
> > > > > > > > > Hi everyone, Thank you all for your interest in this
> DISCUSS.
> > > > > > > > >
> > > > > > > > > @Shammon
> > > > > > > > > > How to handle a catalog with the same name that exists
> for
> > > both
> > > > > of
> > > > > > > > them?
> > > > > > > > >
> > > > > > > > > I believe this is a crucial point. Based on my current
> > personal
> > > > > > > > > understanding, the Map<String, Catalog> catalogs will serve
> > as
> > > a
> > > > > > cache
> > > > > > > > > for instantiated catalogs and have the highest priority.
> > > > > > > > >
> > > > > > > > > There are three methods that can affect the Map<String,
> > > Catalog>
> > > > > > > > catalogs:
> > > > > > > > >
> > > > > > > > > 1. registerCatalog(String catalogName, Catalog catalog)
> > > > > > > > >
> > > > > > > > > This method puts the catalog instance into the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs.
> > > > > > > > >
> > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > >
> > > > > > > > > This method removes the catalog instance corresponding to
> > > > > catalogName
> > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > >
> > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > >
> > > > > > > > > This method first retrieves the corresponding catalog
> > instance
> > > > from
> > > > > > > > > the Map<String, Catalog> catalogs. If the catalog does not
> > > exist,
> > > > > it
> > > > > > > > > retrieves the corresponding configuration from the
> > > CatalogStore,
> > > > > > > > > initializes it, and puts the initialized Catalog instance
> > into
> > > > the
> > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > >
> > > > > > > > > The following two methods only modify the configuration in
> > the
> > > > > > > > > CatalogStore:
> > > > > > > > >
> > > > > > > > > 1. addCatalog(String catalogName, Map<String, String>
> > > properties)
> > > > > > > > >
> > > > > > > > > This method saves the properties to the catalogStore and
> > checks
> > > > > > > > > whether there is a catalogName with the same name.
> > > > > > > > >
> > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > This method removes the specified configuration of the
> > > specified
> > > > > > > > > catalogName in the catalogStore.
> > > > > > > > >
> > > > > > > > > The following are possible conflict scenarios:
> > > > > > > > >
> > > > > > > > > 1. When the corresponding catalogName already exists in the
> > > > > > > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > > > > > > registerCatalog(String catalogName, Catalog catalog) method
> > can
> > > > > > > > > succeed and be directly saved to the Map<String, Catalog>
> > > > catalogs.
> > > > > > > > >
> > > > > > > > > 2. When the corresponding catalogName already exists in
> both
> > > the
> > > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > > registerCatalog(String
> > > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > > >
> > > > > > > > > 3. When the corresponding catalogName already exists in the
> > > > > > > > > Map<String, Catalog>, the addCatalog(String catalogName,
> > > > > Map<String,
> > > > > > > > > String> properties) method can directly save the properties
> > to
> > > > the
> > > > > > > > > catalogStore, but the getCatalog(String catalogName) method
> > > will
> > > > > not
> > > > > > > > > use the new properties for initialization because the
> > > > corresponding
> > > > > > > > > catalog instance already exists in catalogs and will be
> > > > > prioritized.
> > > > > > > > > Therefore, using the unregisterCatalog(String catalogName)
> > > method
> > > > > to
> > > > > > > > > remove the instance corresponding to the original
> catalogName
> > > is
> > > > > > > > > necessary.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > I think it will confuse users that
> `registerCatalog(String
> > > > > > > > > catalogName,Catalog catalog)` in the `Map<String, Catalog>
> > > > > catalogs`
> > > > > > > and
> > > > > > > > > `registerCatalog(String catalogName, Map<String, String>
> > > > > properties)
> > > > > > > > >
> > > > > > > > > This could potentially lead to confusion. I suggest
> changing
> > > the
> > > > > > > > > method name, perhaps to addCatalog(String catalogName,
> > > > Map<String,
> > > > > > > > > String> properties), as previously mentioned
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > @Hang
> > > > > > > > > > add `registerCatalog(String catalogName,Catalog catalog,
> > > > > > > > > boolean lazyInit)` method
> > > > > > > > >
> > > > > > > > > Since a catalog is already an instance, adding the
> "lazyInit"
> > > > > > > > > parameter to the registerCatalog(String catalogName,
> Catalog
> > > > > catalog)
> > > > > > > > > method may not necessarily result in lazy initialization.
> > > > > > > > >
> > > > > > > > > > Do we need to think about encryption
> > > > > > > > >
> > > > > > > > > I think encryption is necessary, but perhaps the encryption
> > > logic
> > > > > > > > > should be implemented in the CatalogStore when the data is
> > > > actually
> > > > > > > > > saved. For instance, the FileCatalogStore could encrypt the
> > > data
> > > > > when
> > > > > > > > > it is saved to a file, while the MemoryCatalogStore does
> not
> > > > > require
> > > > > > > > > encryption.
> > > > > > > > >
> > > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > > >
> > > > > > > > > I think it is necessary to have a MemoryCatalogStore as the
> > > > default
> > > > > > > > > implementation, which saves Catalog configurations in
> memory.
> > > > > > > > > Otherwise, if we want to implement asynchronous loading of
> > > > Catalog,
> > > > > > we
> > > > > > > > > would need to introduce additional cache.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > @Xianxun
> > > > > > > > >
> > > > > > > > > > What if asynchronous registration failed?
> > > > > > > > >
> > > > > > > > > This is also a critical concern. When executing DDL, DQL,
> or
> > > DML
> > > > > > > > > statements that reference a specific Catalog, the Catalog
> > > > instance
> > > > > > > > > will be initialized if it has not been initialized yet. If
> > the
> > > > > > > > > initialization process fails, these statements will not be
> > > > executed
> > > > > > > > > successfully.
> > > > > > > > >
> > > > > > > > > > nt. The Map<String, Catalog> catalogs and CatalogStore
> > > > > catalogstore
> > > > > > > in
> > > > > > > > > the CatalogManager all have the same catalog name, but
> > > > correponding
> > > > > > to
> > > > > > > > > different catalog instances.
> > > > > > > > >
> > > > > > > > > This issue can be resolved by referring to the previous
> > > > responses.
> > > > > > The
> > > > > > > > > key principle is that the Catalog that has been most
> recently
> > > > used
> > > > > > > > > will be given priority for subsequent use.
> > > > > > > > >
> > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > > yesorno828423@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for driving this Flip, I do believe this Flip
> could
> > be
> > > > > > helpful
> > > > > > > > > for users. The firm I work for also manages a lot of
> > catalogs,
> > > > and
> > > > > > the
> > > > > > > > > submission of tasks becomes slow because of loading a
> number
> > of
> > > > > > > catalogs.
> > > > > > > > > We obtain the catalogs in the user's Flink SQL through
> > regular
> > > > > > > expression
> > > > > > > > > to avoid loading all catalogs to improve the speed of task
> > > > > > submission.
> > > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > > >
> > > > > > > > > > 1. When creating a catalog with CREATE CATALOG, the
> > > > asynchronous
> > > > > > > > > registration method is used by default. What if
> asynchronous
> > > > > > > registration
> > > > > > > > > failed? And how to implement asynchronous registration?
> > > > > > > > > >
> > > > > > > > > > 2. I also have the same question with Shammon’s first
> > > comment.
> > > > > The
> > > > > > > > > Map<String, Catalog> catalogs and CatalogStore catalogstore
> > in
> > > > the
> > > > > > > > > CatalogManager all have the same catalog name, but
> > correponding
> > > > to
> > > > > > > > > different catalog instances. For example: catalog1->
> > > jdbcCatalog
> > > > in
> > > > > > the
> > > > > > > > > catalogs, but catalog1 -> hiveCatalog in the catalogstore.
> > > Would
> > > > > this
> > > > > > > > case
> > > > > > > > > happen? Maybe we should define `catalogs` and
> `catalogstore`
> > > more
> > > > > > > > clearly.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > >
> > > > > > > > > > Xianxun
> > > > > > > > > >
> > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com>
> 写道：
> > > > > > > > > > >
> > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Feng,
Thanks for working on the FLIP. There are still some NIT issues in the FLIP
like:
1. Optional<CatalogStore> catalogStore has been used as CatalogStore
instead of Optional in the code example. It should be fine to use it as
pseudo code for now and update it after you submit the PR.
2. addCatalog(...) is still used somewhere in the rejected section which
should be persistContext(...) to keep it consistent.
Speaking of the conflict issues in the multi-instance scenarios, I am not
sure if this is the intended behaviour. If Map<String, Catalog> catalogs is
used as a cache, it should be invalid, once the related catalog has been
removed from the CatalogStore by another instance. Did I miss something?
Best regards,
Jing
On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <ji...@gmail.com> wrote:
> Hi Jing,Shammon
> Thanks for your reply.
>
> @Jing
>
> > How about persistCatalog()?
> I think this is a good function name, I have updated it in the
> documentation.
>
> > Some common cache features should be implemented
> Thank you for the suggestion. If alternative 1 turns out to be more
> appropriate later, I will improve this part of the content.
>
> > As the above discussion moves forward, the option 2 solution looks more
> like a replacement of option 1
> Yes, after discussing with Shammon offline, we think that solution 2 might
> be more suitable and also avoid any inconsistency issues.
>
> > There are some inconsistent descriptions in the content. Would you like
> to clean them up?
> I will do my best to improve the document and appreciate your suggestions.
>
>
>
> @Shammon
> > can you put the unselected option in `Rejected Alternatives`
> Sure, I have moved it to the `Rejected Alternatives`.
>
>
>
> Best
> Feng
>
>
>
> On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com> wrote:
>
> > Hi Feng
> >
> > Thanks for your update.
> >
> > I found there are two options in `Proposed Changes`, can you put the
> > unselected option in `Rejected Alternatives`? I think this may help us
> > better understand your proposal
> >
> >
> > Best,
> > Shammon FY
> >
> >
> > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > Hi Feng,
> > >
> > > Thanks for raising this FLIP. I am still confused after completely
> > reading
> > > the thread with following questions:
> > >
> > > 1. Naming confusion - registerCatalog() and addCatalog() have no big
> > > difference based on their names. One of them is responsible for data
> > > persistence. How about persistCatalog()?
> > > 2. As you mentioned that Map<String, Catalog> catalogs is used as a
> cache
> > > and catalogStore is used for data persistence. I would suggest
> describing
> > > their purpose conceptually and clearly in the FLIP. Some common cache
> > > features should be implemented, i.e. data in the cache and the store
> > should
> > > be consistent. Same Catalog instance should be found in the store and
> in
> > > the cache(either it has been initialized or it will be lazy
> initialized)
> > > for the same catalog name. The consistency will be taken care of while
> > > updating the catalog.
> > > 3. As the above discussion moves forward, the option 2 solution looks
> > more
> > > like a replacement of option 1, because, afaiu, issues mentioned
> > > previously with option 1 are not solved yet. Do you still want to
> propose
> > > both options and ask for suggestions for both of them?
> > > 4. After you updated the FLIP, there are some inconsistent descriptions
> > in
> > > the content. Would you like to clean them up? Thanks!
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <ji...@gmail.com> wrote:
> > >
> > > > hi Shammon
> > > >
> > > > Thank you for your response, and I completely agree with your point
> of
> > > > view.
> > > > Initially, I may have over complicated the whole issue. First and
> > > foremost,
> > > > we need to consider the persistence of the Catalog's Configuration.
> > > > If we only need to provide persistence for Catalog Configuration, we
> > can
> > > > add a toConfiguration method to the Catalog interface.
> > > > This method can convert a Catalog instance to a Map<String, String>
> > > > properties, and the default implementation will throw an exception.
> > > >
> > > > public interface Catalog {
> > > > /**
> > > > * Returns a map containing the properties of the catalog object.
> > > > *
> > > > * @return a map containing the properties of the catalog object
> > > > * @throws UnsupportedOperationException if the implementing class
> > does
> > > > not override
> > > > * the default implementation of this method
> > > > */
> > > > default Map<String, String> toProperties() {
> > > > throw new UnsupportedOperationException("Please implement
> > > toProperties
> > > > for this catalog");
> > > > }
> > > > }
> > > >
> > > > The specific process is as follows:
> > > >
> > > > 1. If the user has configured a CatalogStore, the toConfiguration()
> > > method
> > > > will be called when registering a Catalog instance with
> > > > registerCatalog(String catalogName, Catalog catalog). The Catalog
> > > instance
> > > > will be converted to a Map<String, String> properties and saved to
> the
> > > > CatalogStore. If some Catalog instances do not implement this method,
> > an
> > > > exception will be thrown.
> > > >
> > > > 2. If the user does not use a CatalogStore, the toConfiguration()
> > method
> > > > will not be called, ensuring consistency with the original process.
> > > >
> > > > 3. Saving both the Map<String, Catalog> catalogs and the CatalogStore
> > at
> > > > the same time can also avoid conflicts
> > > >
> > > >
> > > > For lazy initialization:
> > > > we can start from the Catalog itself and provide a dedicated Catalog
> > > > implementation for lazy loading, such as ConfigurationCatalog
> > > >
> > > > public class ConfigurationCatalog implements Catalog {
> > > > ConfigurationCatalog(Map<String, String> properties) {
> > > > }
> > > > }
> > > >
> > > > I have added this design to the FLIP.
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <zj...@gmail.com>
> wrote:
> > > >
> > > > > Hi Feng
> > > > >
> > > > > Thanks for your answer.
> > > > >
> > > > > I have no questions about the functionality of `CatalogStore`, but
> > the
> > > > > different operations of multiple `registerCatalog` or
> `storeCatalog`
> > in
> > > > > `CatalogManager`. The implementation in Trino is also the same, the
> > > > > `CatalogManager` in trino only has one `createCatalog`, which will
> > save
> > > > the
> > > > > catalog to memory and `CatalogStore`.
> > > > >
> > > > > I think we don't need to add another `registerCatalog` or
> > `addCatalog`
> > > to
> > > > > save a catalog in `Map<String, Catalog> catalogs` or `CatalogStore
> > > > > catalogStore`. As you mentioned before, the `Map<String, Catalog>
> > > > > catalogs` is a cache in `CatalogManager`. How about saving the
> > catalog
> > > in
> > > > > `Map<String, Catalog> catalogs` and `CatalogStore catalogStore`
> > > together
> > > > > when it is registered or added in `CatalogManager`?
> > > > >
> > > > > Best,
> > > > > Shammon FY
> > > > >
> > > > >
> > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <ji...@gmail.com>
> > wrote:
> > > > >
> > > > > > Thank you for your reply. I am very sorry for the
> misunderstanding
> > > > caused
> > > > > > by my deviation from the original discussion.
> > > > > >
> > > > > > @Shammon
> > > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > > Yes, there was indeed such a discussion before. However, in
> > > designing
> > > > > the
> > > > > > whole solution, I found that the logic of CatalogManager itself
> > > doesn't
> > > > > > need to change much. *We cannot only persist Catalog instances
> > > > > themselves*,
> > > > > > so exposing only registerCatalog(String catalogName, Catalog
> > catalog)
> > > > > might
> > > > > > not be enough to save Catalogs, because in the end we still need
> to
> > > > save
> > > > > > the configurations corresponding to the Catalog instances.
> > > Therefore,
> > > > I
> > > > > > decided to introduce the CatalogStore interface for configuration
> > > > > > persistence. Regarding this part, I also took inspiration from
> > > Trino's
> > > > > > implementation[1].
> > > > > >
> > > > > >
> > > > > > @yuxia
> > > > > >
> > > > > > > 1: The mechanism of handling catalog with the same name looks a
> > > > little
> > > > > of
> > > > > > complex to me.
> > > > > > Thank you for the suggestion. I will provide a detailed
> description
> > > and
> > > > > > code examples for this part, and add it to the FLIP.
> > > > > >
> > > > > > > 2: TBH, the method name `addCatalog` still looks confused to
> me.
> > > > IIUC,
> > > > > > this method is for storing catalog to CatalogStore, how about
> > > renaming
> > > > it
> > > > > > to `storeCatalog`? It's very personal opinion, you can decide to
> > take
> > > > it
> > > > > or
> > > > > > not by your self.
> > > > > > StoreCatalog looks more intuitive to me. I don't see any problem
> > with
> > > > it.
> > > > > >
> > > > > > > 3.For CREATE CATALOG statement, which method will be called?
> > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether user can
> > > add a
> > > > > > catalog to store with SQL stement.
> > > > > > For CREATE CATALOG, my original design was to add it directly to
> > the
> > > > > > CatalogStore, but this would disrupt the existing logic.
> > Therefore, I
> > > > > think
> > > > > > we can do both: save the configuration to the CatalogStore and
> > > > initialize
> > > > > > the Catalog instance at the same time
> > > > > >
> > > > > > > 3. Is it really neccessary to provide a default implmentation
> for
> > > > > > interface `CatalogStoreFactory`?
> > > > > > I think it is necessary, otherwise we would need to introduce an
> > > > > additional
> > > > > > Map to store the configuration for lazy loading.
> > > > > >
> > > > > > > 4: About asynchronous registration for catalog.
> > > > > > I don't think registerCatalog(String catalogName, Catalog
> catalog)
> > > can
> > > > be
> > > > > > made into an asynchronous interface because Catalog is already an
> > > > > instance.
> > > > > >
> > > > > > > It looks more like lazy initialization for catalog than
> > > asynchronous
> > > > > > registration, right?
> > > > > > Indeed, my description was inaccurate. It should be lazy
> > registration
> > > > > > instead of asynchronous registration. I have already updated the
> > > title
> > > > of
> > > > > > the FLIP.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > [1].
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zj...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Feng
> > > > > > >
> > > > > > > I think if there is a `registerCatalog` method in
> > `CatalogManager`,
> > > > it
> > > > > > will
> > > > > > > confuse users whether a method named `addCatalog` or
> > `storeCatalog`
> > > > is
> > > > > > > added.
> > > > > > >
> > > > > > > And as you mentioned, the memory catalog is a `cache`, I think
> > the
> > > > > > concept
> > > > > > > of `cache` should not be exposed to users.
> > > > > > >
> > > > > > > I found there is a pre-discussion [1] for this FLIP. Please
> > correct
> > > > me
> > > > > if
> > > > > > > I'm wrong, IIUC, the conclusion of that discussion is to use
> > > > > > > `CatalogManager` as an interface and implement it for different
> > > > stores
> > > > > > such
> > > > > > > as memory, file and external system.
> > > > > > >
> > > > > > > I think there is a gap between the current FLIP design and that
> > > > > > conclusion.
> > > > > > > What about the proposal of the discussion in thread [1] ?
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Shammon FY
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > I have few comments:
> > > > > > > > 1: The mechanism of handling catalog with the same name
> looks a
> > > > > little
> > > > > > of
> > > > > > > > complex to me. I think it'll be better to explain it in the
> > java
> > > > doc
> > > > > of
> > > > > > > > these methods and give a brief example in this FLIP.
> > > > > > > >
> > > > > > > > 2: TBH, the method name `addCatalog` still looks confused to
> > me.
> > > > > IIUC,
> > > > > > > > this method is for storing catalog to CatalogStore, how about
> > > > > renaming
> > > > > > it
> > > > > > > > to `storeCatalog`? It's very personal opinion, you can decide
> > to
> > > > take
> > > > > > it
> > > > > > > or
> > > > > > > > not by your self.
> > > > > > > >
> > > > > > > > 3: For CREATE CATALOG statement, which method will be called?
> > > > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether user
> > can
> > > > > add a
> > > > > > > > catalog to store with SQL stement.
> > > > > > > >
> > > > > > > >
> > > > > > > > 3: Is it really neccessary to provide a default implmentation
> > for
> > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > >
> > > > > > > >
> > > > > > > > 4: About asynchronous registration for catalog.
> > > > > > > >
> > > > > > > > > When creating a catalog with CREATE CATALOG, the
> asynchronous
> > > > > > > > registration method is used by default.
> > > > > > > > If asynchronous registration is the default behavior, it
> there
> > > any
> > > > > way
> > > > > > > > that user can switch to synchronous registration just like
> > > before?
> > > > > > > > Will both method `addCatalog` and `registerCatalog` be
> > > asynchronous
> > > > > > > > registration?
> > > > > > > >
> > > > > > > > IIUC, in asynchronous registration, it may well that CREATE
> > > CATALOG
> > > > > > > > executes successfully, but then the following CREATE TABLE
> > > > statement
> > > > > > will
> > > > > > > > fail for the catalog fail to open.
> > > > > > > > I think it's a break change which should be highlighted in
> this
> > > > FLIP,
> > > > > > may
> > > > > > > > be in compatibility part.
> > > > > > > >
> > > > > > > >
> > > > > > > > BTW, by saying asynchronous registration, I would like to
> > expect
> > > > > there
> > > > > > > > will be an executor to open or register catalog in the
> > > background,
> > > > > but
> > > > > > > from
> > > > > > > > your previous comments,
> > > > > > > > "the Catalog instance will be initialized if it has not been
> > > > > > initialized
> > > > > > > > yet. If the initialization process fails, these statements
> will
> > > not
> > > > > be
> > > > > > > > executed successfully."
> > > > > > > > It looks more like lazy initialization for catalog than
> > > > asynchronous
> > > > > > > > registration, right?
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Yuxia
> > > > > > > >
> > > > > > > > ----- 原始邮件 -----
> > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > > configuration
> > > > > > > > and asynchronous registration
> > > > > > > >
> > > > > > > > Hi everyone, Thank you all for your interest in this DISCUSS.
> > > > > > > >
> > > > > > > > @Shammon
> > > > > > > > > How to handle a catalog with the same name that exists for
> > both
> > > > of
> > > > > > > them?
> > > > > > > >
> > > > > > > > I believe this is a crucial point. Based on my current
> personal
> > > > > > > > understanding, the Map<String, Catalog> catalogs will serve
> as
> > a
> > > > > cache
> > > > > > > > for instantiated catalogs and have the highest priority.
> > > > > > > >
> > > > > > > > There are three methods that can affect the Map<String,
> > Catalog>
> > > > > > > catalogs:
> > > > > > > >
> > > > > > > > 1. registerCatalog(String catalogName, Catalog catalog)
> > > > > > > >
> > > > > > > > This method puts the catalog instance into the Map<String,
> > > Catalog>
> > > > > > > > catalogs.
> > > > > > > >
> > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > >
> > > > > > > > This method removes the catalog instance corresponding to
> > > > catalogName
> > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > >
> > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > >
> > > > > > > > This method first retrieves the corresponding catalog
> instance
> > > from
> > > > > > > > the Map<String, Catalog> catalogs. If the catalog does not
> > exist,
> > > > it
> > > > > > > > retrieves the corresponding configuration from the
> > CatalogStore,
> > > > > > > > initializes it, and puts the initialized Catalog instance
> into
> > > the
> > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > >
> > > > > > > > The following two methods only modify the configuration in
> the
> > > > > > > > CatalogStore:
> > > > > > > >
> > > > > > > > 1. addCatalog(String catalogName, Map<String, String>
> > properties)
> > > > > > > >
> > > > > > > > This method saves the properties to the catalogStore and
> checks
> > > > > > > > whether there is a catalogName with the same name.
> > > > > > > >
> > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > This method removes the specified configuration of the
> > specified
> > > > > > > > catalogName in the catalogStore.
> > > > > > > >
> > > > > > > > The following are possible conflict scenarios:
> > > > > > > >
> > > > > > > > 1. When the corresponding catalogName already exists in the
> > > > > > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > > > > > registerCatalog(String catalogName, Catalog catalog) method
> can
> > > > > > > > succeed and be directly saved to the Map<String, Catalog>
> > > catalogs.
> > > > > > > >
> > > > > > > > 2. When the corresponding catalogName already exists in both
> > the
> > > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > > registerCatalog(String
> > > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > > >
> > > > > > > > 3. When the corresponding catalogName already exists in the
> > > > > > > > Map<String, Catalog>, the addCatalog(String catalogName,
> > > > Map<String,
> > > > > > > > String> properties) method can directly save the properties
> to
> > > the
> > > > > > > > catalogStore, but the getCatalog(String catalogName) method
> > will
> > > > not
> > > > > > > > use the new properties for initialization because the
> > > corresponding
> > > > > > > > catalog instance already exists in catalogs and will be
> > > > prioritized.
> > > > > > > > Therefore, using the unregisterCatalog(String catalogName)
> > method
> > > > to
> > > > > > > > remove the instance corresponding to the original catalogName
> > is
> > > > > > > > necessary.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > I think it will confuse users that `registerCatalog(String
> > > > > > > > catalogName,Catalog catalog)` in the `Map<String, Catalog>
> > > > catalogs`
> > > > > > and
> > > > > > > > `registerCatalog(String catalogName, Map<String, String>
> > > > properties)
> > > > > > > >
> > > > > > > > This could potentially lead to confusion. I suggest changing
> > the
> > > > > > > > method name, perhaps to addCatalog(String catalogName,
> > > Map<String,
> > > > > > > > String> properties), as previously mentioned
> > > > > > > >
> > > > > > > >
> > > > > > > > @Hang
> > > > > > > > > add `registerCatalog(String catalogName,Catalog catalog,
> > > > > > > > boolean lazyInit)` method
> > > > > > > >
> > > > > > > > Since a catalog is already an instance, adding the "lazyInit"
> > > > > > > > parameter to the registerCatalog(String catalogName, Catalog
> > > > catalog)
> > > > > > > > method may not necessarily result in lazy initialization.
> > > > > > > >
> > > > > > > > > Do we need to think about encryption
> > > > > > > >
> > > > > > > > I think encryption is necessary, but perhaps the encryption
> > logic
> > > > > > > > should be implemented in the CatalogStore when the data is
> > > actually
> > > > > > > > saved. For instance, the FileCatalogStore could encrypt the
> > data
> > > > when
> > > > > > > > it is saved to a file, while the MemoryCatalogStore does not
> > > > require
> > > > > > > > encryption.
> > > > > > > >
> > > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > > >
> > > > > > > > I think it is necessary to have a MemoryCatalogStore as the
> > > default
> > > > > > > > implementation, which saves Catalog configurations in memory.
> > > > > > > > Otherwise, if we want to implement asynchronous loading of
> > > Catalog,
> > > > > we
> > > > > > > > would need to introduce additional cache.
> > > > > > > >
> > > > > > > >
> > > > > > > > @Xianxun
> > > > > > > >
> > > > > > > > > What if asynchronous registration failed?
> > > > > > > >
> > > > > > > > This is also a critical concern. When executing DDL, DQL, or
> > DML
> > > > > > > > statements that reference a specific Catalog, the Catalog
> > > instance
> > > > > > > > will be initialized if it has not been initialized yet. If
> the
> > > > > > > > initialization process fails, these statements will not be
> > > executed
> > > > > > > > successfully.
> > > > > > > >
> > > > > > > > > nt. The Map<String, Catalog> catalogs and CatalogStore
> > > > catalogstore
> > > > > > in
> > > > > > > > the CatalogManager all have the same catalog name, but
> > > correponding
> > > > > to
> > > > > > > > different catalog instances.
> > > > > > > >
> > > > > > > > This issue can be resolved by referring to the previous
> > > responses.
> > > > > The
> > > > > > > > key principle is that the Catalog that has been most recently
> > > used
> > > > > > > > will be given priority for subsequent use.
> > > > > > > >
> > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > > yesorno828423@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > Thanks for driving this Flip, I do believe this Flip could
> be
> > > > > helpful
> > > > > > > > for users. The firm I work for also manages a lot of
> catalogs,
> > > and
> > > > > the
> > > > > > > > submission of tasks becomes slow because of loading a number
> of
> > > > > > catalogs.
> > > > > > > > We obtain the catalogs in the user's Flink SQL through
> regular
> > > > > > expression
> > > > > > > > to avoid loading all catalogs to improve the speed of task
> > > > > submission.
> > > > > > > > After reading flip-295, I have some questions:
> > > > > > > > >
> > > > > > > > > 1. When creating a catalog with CREATE CATALOG, the
> > > asynchronous
> > > > > > > > registration method is used by default. What if asynchronous
> > > > > > registration
> > > > > > > > failed? And how to implement asynchronous registration?
> > > > > > > > >
> > > > > > > > > 2. I also have the same question with Shammon’s first
> > comment.
> > > > The
> > > > > > > > Map<String, Catalog> catalogs and CatalogStore catalogstore
> in
> > > the
> > > > > > > > CatalogManager all have the same catalog name, but
> correponding
> > > to
> > > > > > > > different catalog instances. For example: catalog1->
> > jdbcCatalog
> > > in
> > > > > the
> > > > > > > > catalogs, but catalog1 -> hiveCatalog in the catalogstore.
> > Would
> > > > this
> > > > > > > case
> > > > > > > > happen? Maybe we should define `catalogs` and `catalogstore`
> > more
> > > > > > > clearly.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > > Xianxun
> > > > > > > > >
> > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> > > > > > > > > >
> > > > > > > > > > MemoryCatalogStore
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Jing,Shammon
Thanks for your reply.
@Jing
> How about persistCatalog()?
I think this is a good function name, I have updated it in the
documentation.
> Some common cache features should be implemented
Thank you for the suggestion. If alternative 1 turns out to be more
appropriate later, I will improve this part of the content.
> As the above discussion moves forward, the option 2 solution looks more
like a replacement of option 1
Yes, after discussing with Shammon offline, we think that solution 2 might
be more suitable and also avoid any inconsistency issues.
> There are some inconsistent descriptions in the content. Would you like
to clean them up?
I will do my best to improve the document and appreciate your suggestions.
@Shammon
> can you put the unselected option in `Rejected Alternatives`
Sure, I have moved it to the `Rejected Alternatives`.
Best
Feng
On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <zj...@gmail.com> wrote:
> Hi Feng
>
> Thanks for your update.
>
> I found there are two options in `Proposed Changes`, can you put the
> unselected option in `Rejected Alternatives`? I think this may help us
> better understand your proposal
>
>
> Best,
> Shammon FY
>
>
> On Thu, Apr 13, 2023 at 4:49 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi Feng,
> >
> > Thanks for raising this FLIP. I am still confused after completely
> reading
> > the thread with following questions:
> >
> > 1. Naming confusion - registerCatalog() and addCatalog() have no big
> > difference based on their names. One of them is responsible for data
> > persistence. How about persistCatalog()?
> > 2. As you mentioned that Map<String, Catalog> catalogs is used as a cache
> > and catalogStore is used for data persistence. I would suggest describing
> > their purpose conceptually and clearly in the FLIP. Some common cache
> > features should be implemented, i.e. data in the cache and the store
> should
> > be consistent. Same Catalog instance should be found in the store and in
> > the cache(either it has been initialized or it will be lazy initialized)
> > for the same catalog name. The consistency will be taken care of while
> > updating the catalog.
> > 3. As the above discussion moves forward, the option 2 solution looks
> more
> > like a replacement of option 1, because, afaiu, issues mentioned
> > previously with option 1 are not solved yet. Do you still want to propose
> > both options and ask for suggestions for both of them?
> > 4. After you updated the FLIP, there are some inconsistent descriptions
> in
> > the content. Would you like to clean them up? Thanks!
> >
> > Best regards,
> > Jing
> >
> >
> > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <ji...@gmail.com> wrote:
> >
> > > hi Shammon
> > >
> > > Thank you for your response, and I completely agree with your point of
> > > view.
> > > Initially, I may have over complicated the whole issue. First and
> > foremost,
> > > we need to consider the persistence of the Catalog's Configuration.
> > > If we only need to provide persistence for Catalog Configuration, we
> can
> > > add a toConfiguration method to the Catalog interface.
> > > This method can convert a Catalog instance to a Map<String, String>
> > > properties, and the default implementation will throw an exception.
> > >
> > > public interface Catalog {
> > > /**
> > > * Returns a map containing the properties of the catalog object.
> > > *
> > > * @return a map containing the properties of the catalog object
> > > * @throws UnsupportedOperationException if the implementing class
> does
> > > not override
> > > * the default implementation of this method
> > > */
> > > default Map<String, String> toProperties() {
> > > throw new UnsupportedOperationException("Please implement
> > toProperties
> > > for this catalog");
> > > }
> > > }
> > >
> > > The specific process is as follows:
> > >
> > > 1. If the user has configured a CatalogStore, the toConfiguration()
> > method
> > > will be called when registering a Catalog instance with
> > > registerCatalog(String catalogName, Catalog catalog). The Catalog
> > instance
> > > will be converted to a Map<String, String> properties and saved to the
> > > CatalogStore. If some Catalog instances do not implement this method,
> an
> > > exception will be thrown.
> > >
> > > 2. If the user does not use a CatalogStore, the toConfiguration()
> method
> > > will not be called, ensuring consistency with the original process.
> > >
> > > 3. Saving both the Map<String, Catalog> catalogs and the CatalogStore
> at
> > > the same time can also avoid conflicts
> > >
> > >
> > > For lazy initialization:
> > > we can start from the Catalog itself and provide a dedicated Catalog
> > > implementation for lazy loading, such as ConfigurationCatalog
> > >
> > > public class ConfigurationCatalog implements Catalog {
> > > ConfigurationCatalog(Map<String, String> properties) {
> > > }
> > > }
> > >
> > > I have added this design to the FLIP.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <zj...@gmail.com> wrote:
> > >
> > > > Hi Feng
> > > >
> > > > Thanks for your answer.
> > > >
> > > > I have no questions about the functionality of `CatalogStore`, but
> the
> > > > different operations of multiple `registerCatalog` or `storeCatalog`
> in
> > > > `CatalogManager`. The implementation in Trino is also the same, the
> > > > `CatalogManager` in trino only has one `createCatalog`, which will
> save
> > > the
> > > > catalog to memory and `CatalogStore`.
> > > >
> > > > I think we don't need to add another `registerCatalog` or
> `addCatalog`
> > to
> > > > save a catalog in `Map<String, Catalog> catalogs` or `CatalogStore
> > > > catalogStore`. As you mentioned before, the `Map<String, Catalog>
> > > > catalogs` is a cache in `CatalogManager`. How about saving the
> catalog
> > in
> > > > `Map<String, Catalog> catalogs` and `CatalogStore catalogStore`
> > together
> > > > when it is registered or added in `CatalogManager`?
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > >
> > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <ji...@gmail.com>
> wrote:
> > > >
> > > > > Thank you for your reply. I am very sorry for the misunderstanding
> > > caused
> > > > > by my deviation from the original discussion.
> > > > >
> > > > > @Shammon
> > > > > > I found there is a pre-discussion [1] for this FLIP
> > > > > Yes, there was indeed such a discussion before. However, in
> > designing
> > > > the
> > > > > whole solution, I found that the logic of CatalogManager itself
> > doesn't
> > > > > need to change much. *We cannot only persist Catalog instances
> > > > themselves*,
> > > > > so exposing only registerCatalog(String catalogName, Catalog
> catalog)
> > > > might
> > > > > not be enough to save Catalogs, because in the end we still need to
> > > save
> > > > > the configurations corresponding to the Catalog instances.
> > Therefore,
> > > I
> > > > > decided to introduce the CatalogStore interface for configuration
> > > > > persistence. Regarding this part, I also took inspiration from
> > Trino's
> > > > > implementation[1].
> > > > >
> > > > >
> > > > > @yuxia
> > > > >
> > > > > > 1: The mechanism of handling catalog with the same name looks a
> > > little
> > > > of
> > > > > complex to me.
> > > > > Thank you for the suggestion. I will provide a detailed description
> > and
> > > > > code examples for this part, and add it to the FLIP.
> > > > >
> > > > > > 2: TBH, the method name `addCatalog` still looks confused to me.
> > > IIUC,
> > > > > this method is for storing catalog to CatalogStore, how about
> > renaming
> > > it
> > > > > to `storeCatalog`? It's very personal opinion, you can decide to
> take
> > > it
> > > > or
> > > > > not by your self.
> > > > > StoreCatalog looks more intuitive to me. I don't see any problem
> with
> > > it.
> > > > >
> > > > > > 3.For CREATE CATALOG statement, which method will be called?
> > > > > `registerCatalog` or `addCatalog`? I'm wondering whether user can
> > add a
> > > > > catalog to store with SQL stement.
> > > > > For CREATE CATALOG, my original design was to add it directly to
> the
> > > > > CatalogStore, but this would disrupt the existing logic.
> Therefore, I
> > > > think
> > > > > we can do both: save the configuration to the CatalogStore and
> > > initialize
> > > > > the Catalog instance at the same time
> > > > >
> > > > > > 3. Is it really neccessary to provide a default implmentation for
> > > > > interface `CatalogStoreFactory`?
> > > > > I think it is necessary, otherwise we would need to introduce an
> > > > additional
> > > > > Map to store the configuration for lazy loading.
> > > > >
> > > > > > 4: About asynchronous registration for catalog.
> > > > > I don't think registerCatalog(String catalogName, Catalog catalog)
> > can
> > > be
> > > > > made into an asynchronous interface because Catalog is already an
> > > > instance.
> > > > >
> > > > > > It looks more like lazy initialization for catalog than
> > asynchronous
> > > > > registration, right?
> > > > > Indeed, my description was inaccurate. It should be lazy
> registration
> > > > > instead of asynchronous registration. I have already updated the
> > title
> > > of
> > > > > the FLIP.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > [1].
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > >
> > > > >
> > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zj...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Feng
> > > > > >
> > > > > > I think if there is a `registerCatalog` method in
> `CatalogManager`,
> > > it
> > > > > will
> > > > > > confuse users whether a method named `addCatalog` or
> `storeCatalog`
> > > is
> > > > > > added.
> > > > > >
> > > > > > And as you mentioned, the memory catalog is a `cache`, I think
> the
> > > > > concept
> > > > > > of `cache` should not be exposed to users.
> > > > > >
> > > > > > I found there is a pre-discussion [1] for this FLIP. Please
> correct
> > > me
> > > > if
> > > > > > I'm wrong, IIUC, the conclusion of that discussion is to use
> > > > > > `CatalogManager` as an interface and implement it for different
> > > stores
> > > > > such
> > > > > > as memory, file and external system.
> > > > > >
> > > > > > I think there is a gap between the current FLIP design and that
> > > > > conclusion.
> > > > > > What about the proposal of the discussion in thread [1] ?
> > > > > >
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Shammon FY
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> luoyuxia@alumni.sjtu.edu.cn>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > I have few comments:
> > > > > > > 1: The mechanism of handling catalog with the same name looks a
> > > > little
> > > > > of
> > > > > > > complex to me. I think it'll be better to explain it in the
> java
> > > doc
> > > > of
> > > > > > > these methods and give a brief example in this FLIP.
> > > > > > >
> > > > > > > 2: TBH, the method name `addCatalog` still looks confused to
> me.
> > > > IIUC,
> > > > > > > this method is for storing catalog to CatalogStore, how about
> > > > renaming
> > > > > it
> > > > > > > to `storeCatalog`? It's very personal opinion, you can decide
> to
> > > take
> > > > > it
> > > > > > or
> > > > > > > not by your self.
> > > > > > >
> > > > > > > 3: For CREATE CATALOG statement, which method will be called?
> > > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether user
> can
> > > > add a
> > > > > > > catalog to store with SQL stement.
> > > > > > >
> > > > > > >
> > > > > > > 3: Is it really neccessary to provide a default implmentation
> for
> > > > > > > interface `CatalogStoreFactory`?
> > > > > > >
> > > > > > >
> > > > > > > 4: About asynchronous registration for catalog.
> > > > > > >
> > > > > > > > When creating a catalog with CREATE CATALOG, the asynchronous
> > > > > > > registration method is used by default.
> > > > > > > If asynchronous registration is the default behavior, it there
> > any
> > > > way
> > > > > > > that user can switch to synchronous registration just like
> > before?
> > > > > > > Will both method `addCatalog` and `registerCatalog` be
> > asynchronous
> > > > > > > registration?
> > > > > > >
> > > > > > > IIUC, in asynchronous registration, it may well that CREATE
> > CATALOG
> > > > > > > executes successfully, but then the following CREATE TABLE
> > > statement
> > > > > will
> > > > > > > fail for the catalog fail to open.
> > > > > > > I think it's a break change which should be highlighted in this
> > > FLIP,
> > > > > may
> > > > > > > be in compatibility part.
> > > > > > >
> > > > > > >
> > > > > > > BTW, by saying asynchronous registration, I would like to
> expect
> > > > there
> > > > > > > will be an executor to open or register catalog in the
> > background,
> > > > but
> > > > > > from
> > > > > > > your previous comments,
> > > > > > > "the Catalog instance will be initialized if it has not been
> > > > > initialized
> > > > > > > yet. If the initialization process fails, these statements will
> > not
> > > > be
> > > > > > > executed successfully."
> > > > > > > It looks more like lazy initialization for catalog than
> > > asynchronous
> > > > > > > registration, right?
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Yuxia
> > > > > > >
> > > > > > > ----- 原始邮件 -----
> > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > configuration
> > > > > > > and asynchronous registration
> > > > > > >
> > > > > > > Hi everyone, Thank you all for your interest in this DISCUSS.
> > > > > > >
> > > > > > > @Shammon
> > > > > > > > How to handle a catalog with the same name that exists for
> both
> > > of
> > > > > > them?
> > > > > > >
> > > > > > > I believe this is a crucial point. Based on my current personal
> > > > > > > understanding, the Map<String, Catalog> catalogs will serve as
> a
> > > > cache
> > > > > > > for instantiated catalogs and have the highest priority.
> > > > > > >
> > > > > > > There are three methods that can affect the Map<String,
> Catalog>
> > > > > > catalogs:
> > > > > > >
> > > > > > > 1. registerCatalog(String catalogName, Catalog catalog)
> > > > > > >
> > > > > > > This method puts the catalog instance into the Map<String,
> > Catalog>
> > > > > > > catalogs.
> > > > > > >
> > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > >
> > > > > > > This method removes the catalog instance corresponding to
> > > catalogName
> > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > >
> > > > > > > 3. getCatalog(String catalogName)
> > > > > > >
> > > > > > > This method first retrieves the corresponding catalog instance
> > from
> > > > > > > the Map<String, Catalog> catalogs. If the catalog does not
> exist,
> > > it
> > > > > > > retrieves the corresponding configuration from the
> CatalogStore,
> > > > > > > initializes it, and puts the initialized Catalog instance into
> > the
> > > > > > > Map<String, Catalog> catalogs.
> > > > > > >
> > > > > > > The following two methods only modify the configuration in the
> > > > > > > CatalogStore:
> > > > > > >
> > > > > > > 1. addCatalog(String catalogName, Map<String, String>
> properties)
> > > > > > >
> > > > > > > This method saves the properties to the catalogStore and checks
> > > > > > > whether there is a catalogName with the same name.
> > > > > > >
> > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > This method removes the specified configuration of the
> specified
> > > > > > > catalogName in the catalogStore.
> > > > > > >
> > > > > > > The following are possible conflict scenarios:
> > > > > > >
> > > > > > > 1. When the corresponding catalogName already exists in the
> > > > > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > > > > registerCatalog(String catalogName, Catalog catalog) method can
> > > > > > > succeed and be directly saved to the Map<String, Catalog>
> > catalogs.
> > > > > > >
> > > > > > > 2. When the corresponding catalogName already exists in both
> the
> > > > > > > CatalogStore and the Map<String, Catalog>, the
> > > registerCatalog(String
> > > > > > > catalogName, Catalog catalog) method will fail.
> > > > > > >
> > > > > > > 3. When the corresponding catalogName already exists in the
> > > > > > > Map<String, Catalog>, the addCatalog(String catalogName,
> > > Map<String,
> > > > > > > String> properties) method can directly save the properties to
> > the
> > > > > > > catalogStore, but the getCatalog(String catalogName) method
> will
> > > not
> > > > > > > use the new properties for initialization because the
> > corresponding
> > > > > > > catalog instance already exists in catalogs and will be
> > > prioritized.
> > > > > > > Therefore, using the unregisterCatalog(String catalogName)
> method
> > > to
> > > > > > > remove the instance corresponding to the original catalogName
> is
> > > > > > > necessary.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > I think it will confuse users that `registerCatalog(String
> > > > > > > catalogName,Catalog catalog)` in the `Map<String, Catalog>
> > > catalogs`
> > > > > and
> > > > > > > `registerCatalog(String catalogName, Map<String, String>
> > > properties)
> > > > > > >
> > > > > > > This could potentially lead to confusion. I suggest changing
> the
> > > > > > > method name, perhaps to addCatalog(String catalogName,
> > Map<String,
> > > > > > > String> properties), as previously mentioned
> > > > > > >
> > > > > > >
> > > > > > > @Hang
> > > > > > > > add `registerCatalog(String catalogName,Catalog catalog,
> > > > > > > boolean lazyInit)` method
> > > > > > >
> > > > > > > Since a catalog is already an instance, adding the "lazyInit"
> > > > > > > parameter to the registerCatalog(String catalogName, Catalog
> > > catalog)
> > > > > > > method may not necessarily result in lazy initialization.
> > > > > > >
> > > > > > > > Do we need to think about encryption
> > > > > > >
> > > > > > > I think encryption is necessary, but perhaps the encryption
> logic
> > > > > > > should be implemented in the CatalogStore when the data is
> > actually
> > > > > > > saved. For instance, the FileCatalogStore could encrypt the
> data
> > > when
> > > > > > > it is saved to a file, while the MemoryCatalogStore does not
> > > require
> > > > > > > encryption.
> > > > > > >
> > > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > > >
> > > > > > > I think it is necessary to have a MemoryCatalogStore as the
> > default
> > > > > > > implementation, which saves Catalog configurations in memory.
> > > > > > > Otherwise, if we want to implement asynchronous loading of
> > Catalog,
> > > > we
> > > > > > > would need to introduce additional cache.
> > > > > > >
> > > > > > >
> > > > > > > @Xianxun
> > > > > > >
> > > > > > > > What if asynchronous registration failed?
> > > > > > >
> > > > > > > This is also a critical concern. When executing DDL, DQL, or
> DML
> > > > > > > statements that reference a specific Catalog, the Catalog
> > instance
> > > > > > > will be initialized if it has not been initialized yet. If the
> > > > > > > initialization process fails, these statements will not be
> > executed
> > > > > > > successfully.
> > > > > > >
> > > > > > > > nt. The Map<String, Catalog> catalogs and CatalogStore
> > > catalogstore
> > > > > in
> > > > > > > the CatalogManager all have the same catalog name, but
> > correponding
> > > > to
> > > > > > > different catalog instances.
> > > > > > >
> > > > > > > This issue can be resolved by referring to the previous
> > responses.
> > > > The
> > > > > > > key principle is that the Catalog that has been most recently
> > used
> > > > > > > will be given priority for subsequent use.
> > > > > > >
> > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > > yesorno828423@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > Thanks for driving this Flip, I do believe this Flip could be
> > > > helpful
> > > > > > > for users. The firm I work for also manages a lot of catalogs,
> > and
> > > > the
> > > > > > > submission of tasks becomes slow because of loading a number of
> > > > > catalogs.
> > > > > > > We obtain the catalogs in the user's Flink SQL through regular
> > > > > expression
> > > > > > > to avoid loading all catalogs to improve the speed of task
> > > > submission.
> > > > > > > After reading flip-295, I have some questions:
> > > > > > > >
> > > > > > > > 1. When creating a catalog with CREATE CATALOG, the
> > asynchronous
> > > > > > > registration method is used by default. What if asynchronous
> > > > > registration
> > > > > > > failed? And how to implement asynchronous registration?
> > > > > > > >
> > > > > > > > 2. I also have the same question with Shammon’s first
> comment.
> > > The
> > > > > > > Map<String, Catalog> catalogs and CatalogStore catalogstore in
> > the
> > > > > > > CatalogManager all have the same catalog name, but correponding
> > to
> > > > > > > different catalog instances. For example: catalog1->
> jdbcCatalog
> > in
> > > > the
> > > > > > > catalogs, but catalog1 -> hiveCatalog in the catalogstore.
> Would
> > > this
> > > > > > case
> > > > > > > happen? Maybe we should define `catalogs` and `catalogstore`
> more
> > > > > > clearly.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Xianxun
> > > > > > > >
> > > > > > > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> > > > > > > > >
> > > > > > > > > MemoryCatalogStore
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Shammon FY <zj...@gmail.com>.
Hi Feng
Thanks for your update.
I found there are two options in `Proposed Changes`, can you put the
unselected option in `Rejected Alternatives`? I think this may help us
better understand your proposal
Best,
Shammon FY
On Thu, Apr 13, 2023 at 4:49 AM Jing Ge <ji...@ververica.com.invalid> wrote:
> Hi Feng,
>
> Thanks for raising this FLIP. I am still confused after completely reading
> the thread with following questions:
>
> 1. Naming confusion - registerCatalog() and addCatalog() have no big
> difference based on their names. One of them is responsible for data
> persistence. How about persistCatalog()?
> 2. As you mentioned that Map<String, Catalog> catalogs is used as a cache
> and catalogStore is used for data persistence. I would suggest describing
> their purpose conceptually and clearly in the FLIP. Some common cache
> features should be implemented, i.e. data in the cache and the store should
> be consistent. Same Catalog instance should be found in the store and in
> the cache(either it has been initialized or it will be lazy initialized)
> for the same catalog name. The consistency will be taken care of while
> updating the catalog.
> 3. As the above discussion moves forward, the option 2 solution looks more
> like a replacement of option 1, because, afaiu, issues mentioned
> previously with option 1 are not solved yet. Do you still want to propose
> both options and ask for suggestions for both of them?
> 4. After you updated the FLIP, there are some inconsistent descriptions in
> the content. Would you like to clean them up? Thanks!
>
> Best regards,
> Jing
>
>
> On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <ji...@gmail.com> wrote:
>
> > hi Shammon
> >
> > Thank you for your response, and I completely agree with your point of
> > view.
> > Initially, I may have over complicated the whole issue. First and
> foremost,
> > we need to consider the persistence of the Catalog's Configuration.
> > If we only need to provide persistence for Catalog Configuration, we can
> > add a toConfiguration method to the Catalog interface.
> > This method can convert a Catalog instance to a Map<String, String>
> > properties, and the default implementation will throw an exception.
> >
> > public interface Catalog {
> > /**
> > * Returns a map containing the properties of the catalog object.
> > *
> > * @return a map containing the properties of the catalog object
> > * @throws UnsupportedOperationException if the implementing class does
> > not override
> > * the default implementation of this method
> > */
> > default Map<String, String> toProperties() {
> > throw new UnsupportedOperationException("Please implement
> toProperties
> > for this catalog");
> > }
> > }
> >
> > The specific process is as follows:
> >
> > 1. If the user has configured a CatalogStore, the toConfiguration()
> method
> > will be called when registering a Catalog instance with
> > registerCatalog(String catalogName, Catalog catalog). The Catalog
> instance
> > will be converted to a Map<String, String> properties and saved to the
> > CatalogStore. If some Catalog instances do not implement this method, an
> > exception will be thrown.
> >
> > 2. If the user does not use a CatalogStore, the toConfiguration() method
> > will not be called, ensuring consistency with the original process.
> >
> > 3. Saving both the Map<String, Catalog> catalogs and the CatalogStore at
> > the same time can also avoid conflicts
> >
> >
> > For lazy initialization:
> > we can start from the Catalog itself and provide a dedicated Catalog
> > implementation for lazy loading, such as ConfigurationCatalog
> >
> > public class ConfigurationCatalog implements Catalog {
> > ConfigurationCatalog(Map<String, String> properties) {
> > }
> > }
> >
> > I have added this design to the FLIP.
> >
> >
> > Best,
> > Feng
> >
> > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <zj...@gmail.com> wrote:
> >
> > > Hi Feng
> > >
> > > Thanks for your answer.
> > >
> > > I have no questions about the functionality of `CatalogStore`, but the
> > > different operations of multiple `registerCatalog` or `storeCatalog` in
> > > `CatalogManager`. The implementation in Trino is also the same, the
> > > `CatalogManager` in trino only has one `createCatalog`, which will save
> > the
> > > catalog to memory and `CatalogStore`.
> > >
> > > I think we don't need to add another `registerCatalog` or `addCatalog`
> to
> > > save a catalog in `Map<String, Catalog> catalogs` or `CatalogStore
> > > catalogStore`. As you mentioned before, the `Map<String, Catalog>
> > > catalogs` is a cache in `CatalogManager`. How about saving the catalog
> in
> > > `Map<String, Catalog> catalogs` and `CatalogStore catalogStore`
> together
> > > when it is registered or added in `CatalogManager`?
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <ji...@gmail.com> wrote:
> > >
> > > > Thank you for your reply. I am very sorry for the misunderstanding
> > caused
> > > > by my deviation from the original discussion.
> > > >
> > > > @Shammon
> > > > > I found there is a pre-discussion [1] for this FLIP
> > > > Yes, there was indeed such a discussion before. However, in
> designing
> > > the
> > > > whole solution, I found that the logic of CatalogManager itself
> doesn't
> > > > need to change much. *We cannot only persist Catalog instances
> > > themselves*,
> > > > so exposing only registerCatalog(String catalogName, Catalog catalog)
> > > might
> > > > not be enough to save Catalogs, because in the end we still need to
> > save
> > > > the configurations corresponding to the Catalog instances.
> Therefore,
> > I
> > > > decided to introduce the CatalogStore interface for configuration
> > > > persistence. Regarding this part, I also took inspiration from
> Trino's
> > > > implementation[1].
> > > >
> > > >
> > > > @yuxia
> > > >
> > > > > 1: The mechanism of handling catalog with the same name looks a
> > little
> > > of
> > > > complex to me.
> > > > Thank you for the suggestion. I will provide a detailed description
> and
> > > > code examples for this part, and add it to the FLIP.
> > > >
> > > > > 2: TBH, the method name `addCatalog` still looks confused to me.
> > IIUC,
> > > > this method is for storing catalog to CatalogStore, how about
> renaming
> > it
> > > > to `storeCatalog`? It's very personal opinion, you can decide to take
> > it
> > > or
> > > > not by your self.
> > > > StoreCatalog looks more intuitive to me. I don't see any problem with
> > it.
> > > >
> > > > > 3.For CREATE CATALOG statement, which method will be called?
> > > > `registerCatalog` or `addCatalog`? I'm wondering whether user can
> add a
> > > > catalog to store with SQL stement.
> > > > For CREATE CATALOG, my original design was to add it directly to the
> > > > CatalogStore, but this would disrupt the existing logic. Therefore, I
> > > think
> > > > we can do both: save the configuration to the CatalogStore and
> > initialize
> > > > the Catalog instance at the same time
> > > >
> > > > > 3. Is it really neccessary to provide a default implmentation for
> > > > interface `CatalogStoreFactory`?
> > > > I think it is necessary, otherwise we would need to introduce an
> > > additional
> > > > Map to store the configuration for lazy loading.
> > > >
> > > > > 4: About asynchronous registration for catalog.
> > > > I don't think registerCatalog(String catalogName, Catalog catalog)
> can
> > be
> > > > made into an asynchronous interface because Catalog is already an
> > > instance.
> > > >
> > > > > It looks more like lazy initialization for catalog than
> asynchronous
> > > > registration, right?
> > > > Indeed, my description was inaccurate. It should be lazy registration
> > > > instead of asynchronous registration. I have already updated the
> title
> > of
> > > > the FLIP.
> > > >
> > > >
> > > >
> > > >
> > > > [1].
> > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > >
> > > >
> > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zj...@gmail.com> wrote:
> > > >
> > > > > Hi Feng
> > > > >
> > > > > I think if there is a `registerCatalog` method in `CatalogManager`,
> > it
> > > > will
> > > > > confuse users whether a method named `addCatalog` or `storeCatalog`
> > is
> > > > > added.
> > > > >
> > > > > And as you mentioned, the memory catalog is a `cache`, I think the
> > > > concept
> > > > > of `cache` should not be exposed to users.
> > > > >
> > > > > I found there is a pre-discussion [1] for this FLIP. Please correct
> > me
> > > if
> > > > > I'm wrong, IIUC, the conclusion of that discussion is to use
> > > > > `CatalogManager` as an interface and implement it for different
> > stores
> > > > such
> > > > > as memory, file and external system.
> > > > >
> > > > > I think there is a gap between the current FLIP design and that
> > > > conclusion.
> > > > > What about the proposal of the discussion in thread [1] ?
> > > > >
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > >
> > > > >
> > > > > Best,
> > > > > Shammon FY
> > > > >
> > > > >
> > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > > > wrote:
> > > > >
> > > > > > Thanks Feng for driving this FLIP.
> > > > > > I have few comments:
> > > > > > 1: The mechanism of handling catalog with the same name looks a
> > > little
> > > > of
> > > > > > complex to me. I think it'll be better to explain it in the java
> > doc
> > > of
> > > > > > these methods and give a brief example in this FLIP.
> > > > > >
> > > > > > 2: TBH, the method name `addCatalog` still looks confused to me.
> > > IIUC,
> > > > > > this method is for storing catalog to CatalogStore, how about
> > > renaming
> > > > it
> > > > > > to `storeCatalog`? It's very personal opinion, you can decide to
> > take
> > > > it
> > > > > or
> > > > > > not by your self.
> > > > > >
> > > > > > 3: For CREATE CATALOG statement, which method will be called?
> > > > > > `registerCatalog` or `addCatalog`? I'm wondering whether user can
> > > add a
> > > > > > catalog to store with SQL stement.
> > > > > >
> > > > > >
> > > > > > 3: Is it really neccessary to provide a default implmentation for
> > > > > > interface `CatalogStoreFactory`?
> > > > > >
> > > > > >
> > > > > > 4: About asynchronous registration for catalog.
> > > > > >
> > > > > > > When creating a catalog with CREATE CATALOG, the asynchronous
> > > > > > registration method is used by default.
> > > > > > If asynchronous registration is the default behavior, it there
> any
> > > way
> > > > > > that user can switch to synchronous registration just like
> before?
> > > > > > Will both method `addCatalog` and `registerCatalog` be
> asynchronous
> > > > > > registration?
> > > > > >
> > > > > > IIUC, in asynchronous registration, it may well that CREATE
> CATALOG
> > > > > > executes successfully, but then the following CREATE TABLE
> > statement
> > > > will
> > > > > > fail for the catalog fail to open.
> > > > > > I think it's a break change which should be highlighted in this
> > FLIP,
> > > > may
> > > > > > be in compatibility part.
> > > > > >
> > > > > >
> > > > > > BTW, by saying asynchronous registration, I would like to expect
> > > there
> > > > > > will be an executor to open or register catalog in the
> background,
> > > but
> > > > > from
> > > > > > your previous comments,
> > > > > > "the Catalog instance will be initialized if it has not been
> > > > initialized
> > > > > > yet. If the initialization process fails, these statements will
> not
> > > be
> > > > > > executed successfully."
> > > > > > It looks more like lazy initialization for catalog than
> > asynchronous
> > > > > > registration, right?
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > > Yuxia
> > > > > >
> > > > > > ----- 原始邮件 -----
> > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > configuration
> > > > > > and asynchronous registration
> > > > > >
> > > > > > Hi everyone, Thank you all for your interest in this DISCUSS.
> > > > > >
> > > > > > @Shammon
> > > > > > > How to handle a catalog with the same name that exists for both
> > of
> > > > > them?
> > > > > >
> > > > > > I believe this is a crucial point. Based on my current personal
> > > > > > understanding, the Map<String, Catalog> catalogs will serve as a
> > > cache
> > > > > > for instantiated catalogs and have the highest priority.
> > > > > >
> > > > > > There are three methods that can affect the Map<String, Catalog>
> > > > > catalogs:
> > > > > >
> > > > > > 1. registerCatalog(String catalogName, Catalog catalog)
> > > > > >
> > > > > > This method puts the catalog instance into the Map<String,
> Catalog>
> > > > > > catalogs.
> > > > > >
> > > > > > 2. unregisterCatalog(String catalogName)
> > > > > >
> > > > > > This method removes the catalog instance corresponding to
> > catalogName
> > > > > > from the Map<String, Catalog> catalogs.
> > > > > >
> > > > > > 3. getCatalog(String catalogName)
> > > > > >
> > > > > > This method first retrieves the corresponding catalog instance
> from
> > > > > > the Map<String, Catalog> catalogs. If the catalog does not exist,
> > it
> > > > > > retrieves the corresponding configuration from the CatalogStore,
> > > > > > initializes it, and puts the initialized Catalog instance into
> the
> > > > > > Map<String, Catalog> catalogs.
> > > > > >
> > > > > > The following two methods only modify the configuration in the
> > > > > > CatalogStore:
> > > > > >
> > > > > > 1. addCatalog(String catalogName, Map<String, String> properties)
> > > > > >
> > > > > > This method saves the properties to the catalogStore and checks
> > > > > > whether there is a catalogName with the same name.
> > > > > >
> > > > > > 2. removeCatalog(String catalogName)
> > > > > > This method removes the specified configuration of the specified
> > > > > > catalogName in the catalogStore.
> > > > > >
> > > > > > The following are possible conflict scenarios:
> > > > > >
> > > > > > 1. When the corresponding catalogName already exists in the
> > > > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > > > registerCatalog(String catalogName, Catalog catalog) method can
> > > > > > succeed and be directly saved to the Map<String, Catalog>
> catalogs.
> > > > > >
> > > > > > 2. When the corresponding catalogName already exists in both the
> > > > > > CatalogStore and the Map<String, Catalog>, the
> > registerCatalog(String
> > > > > > catalogName, Catalog catalog) method will fail.
> > > > > >
> > > > > > 3. When the corresponding catalogName already exists in the
> > > > > > Map<String, Catalog>, the addCatalog(String catalogName,
> > Map<String,
> > > > > > String> properties) method can directly save the properties to
> the
> > > > > > catalogStore, but the getCatalog(String catalogName) method will
> > not
> > > > > > use the new properties for initialization because the
> corresponding
> > > > > > catalog instance already exists in catalogs and will be
> > prioritized.
> > > > > > Therefore, using the unregisterCatalog(String catalogName) method
> > to
> > > > > > remove the instance corresponding to the original catalogName is
> > > > > > necessary.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > I think it will confuse users that `registerCatalog(String
> > > > > > catalogName,Catalog catalog)` in the `Map<String, Catalog>
> > catalogs`
> > > > and
> > > > > > `registerCatalog(String catalogName, Map<String, String>
> > properties)
> > > > > >
> > > > > > This could potentially lead to confusion. I suggest changing the
> > > > > > method name, perhaps to addCatalog(String catalogName,
> Map<String,
> > > > > > String> properties), as previously mentioned
> > > > > >
> > > > > >
> > > > > > @Hang
> > > > > > > add `registerCatalog(String catalogName,Catalog catalog,
> > > > > > boolean lazyInit)` method
> > > > > >
> > > > > > Since a catalog is already an instance, adding the "lazyInit"
> > > > > > parameter to the registerCatalog(String catalogName, Catalog
> > catalog)
> > > > > > method may not necessarily result in lazy initialization.
> > > > > >
> > > > > > > Do we need to think about encryption
> > > > > >
> > > > > > I think encryption is necessary, but perhaps the encryption logic
> > > > > > should be implemented in the CatalogStore when the data is
> actually
> > > > > > saved. For instance, the FileCatalogStore could encrypt the data
> > when
> > > > > > it is saved to a file, while the MemoryCatalogStore does not
> > require
> > > > > > encryption.
> > > > > >
> > > > > > > Do we really need the `MemoryCatalogStore`?
> > > > > >
> > > > > > I think it is necessary to have a MemoryCatalogStore as the
> default
> > > > > > implementation, which saves Catalog configurations in memory.
> > > > > > Otherwise, if we want to implement asynchronous loading of
> Catalog,
> > > we
> > > > > > would need to introduce additional cache.
> > > > > >
> > > > > >
> > > > > > @Xianxun
> > > > > >
> > > > > > > What if asynchronous registration failed?
> > > > > >
> > > > > > This is also a critical concern. When executing DDL, DQL, or DML
> > > > > > statements that reference a specific Catalog, the Catalog
> instance
> > > > > > will be initialized if it has not been initialized yet. If the
> > > > > > initialization process fails, these statements will not be
> executed
> > > > > > successfully.
> > > > > >
> > > > > > > nt. The Map<String, Catalog> catalogs and CatalogStore
> > catalogstore
> > > > in
> > > > > > the CatalogManager all have the same catalog name, but
> correponding
> > > to
> > > > > > different catalog instances.
> > > > > >
> > > > > > This issue can be resolved by referring to the previous
> responses.
> > > The
> > > > > > key principle is that the Catalog that has been most recently
> used
> > > > > > will be given priority for subsequent use.
> > > > > >
> > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> > yesorno828423@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > Thanks for driving this Flip, I do believe this Flip could be
> > > helpful
> > > > > > for users. The firm I work for also manages a lot of catalogs,
> and
> > > the
> > > > > > submission of tasks becomes slow because of loading a number of
> > > > catalogs.
> > > > > > We obtain the catalogs in the user's Flink SQL through regular
> > > > expression
> > > > > > to avoid loading all catalogs to improve the speed of task
> > > submission.
> > > > > > After reading flip-295, I have some questions:
> > > > > > >
> > > > > > > 1. When creating a catalog with CREATE CATALOG, the
> asynchronous
> > > > > > registration method is used by default. What if asynchronous
> > > > registration
> > > > > > failed? And how to implement asynchronous registration?
> > > > > > >
> > > > > > > 2. I also have the same question with Shammon’s first comment.
> > The
> > > > > > Map<String, Catalog> catalogs and CatalogStore catalogstore in
> the
> > > > > > CatalogManager all have the same catalog name, but correponding
> to
> > > > > > different catalog instances. For example: catalog1-> jdbcCatalog
> in
> > > the
> > > > > > catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would
> > this
> > > > > case
> > > > > > happen? Maybe we should define `catalogs` and `catalogstore` more
> > > > > clearly.
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Xianxun
> > > > > > >
> > > > > > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> > > > > > > >
> > > > > > > > MemoryCatalogStore
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Feng,
Thanks for raising this FLIP. I am still confused after completely reading
the thread with following questions:
1. Naming confusion - registerCatalog() and addCatalog() have no big
difference based on their names. One of them is responsible for data
persistence. How about persistCatalog()?
2. As you mentioned that Map<String, Catalog> catalogs is used as a cache
and catalogStore is used for data persistence. I would suggest describing
their purpose conceptually and clearly in the FLIP. Some common cache
features should be implemented, i.e. data in the cache and the store should
be consistent. Same Catalog instance should be found in the store and in
the cache(either it has been initialized or it will be lazy initialized)
for the same catalog name. The consistency will be taken care of while
updating the catalog.
3. As the above discussion moves forward, the option 2 solution looks more
like a replacement of option 1, because, afaiu, issues mentioned
previously with option 1 are not solved yet. Do you still want to propose
both options and ask for suggestions for both of them?
4. After you updated the FLIP, there are some inconsistent descriptions in
the content. Would you like to clean them up? Thanks!
Best regards,
Jing
On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <ji...@gmail.com> wrote:
> hi Shammon
>
> Thank you for your response, and I completely agree with your point of
> view.
> Initially, I may have over complicated the whole issue. First and foremost,
> we need to consider the persistence of the Catalog's Configuration.
> If we only need to provide persistence for Catalog Configuration, we can
> add a toConfiguration method to the Catalog interface.
> This method can convert a Catalog instance to a Map<String, String>
> properties, and the default implementation will throw an exception.
>
> public interface Catalog {
> /**
> * Returns a map containing the properties of the catalog object.
> *
> * @return a map containing the properties of the catalog object
> * @throws UnsupportedOperationException if the implementing class does
> not override
> * the default implementation of this method
> */
> default Map<String, String> toProperties() {
> throw new UnsupportedOperationException("Please implement toProperties
> for this catalog");
> }
> }
>
> The specific process is as follows:
>
> 1. If the user has configured a CatalogStore, the toConfiguration() method
> will be called when registering a Catalog instance with
> registerCatalog(String catalogName, Catalog catalog). The Catalog instance
> will be converted to a Map<String, String> properties and saved to the
> CatalogStore. If some Catalog instances do not implement this method, an
> exception will be thrown.
>
> 2. If the user does not use a CatalogStore, the toConfiguration() method
> will not be called, ensuring consistency with the original process.
>
> 3. Saving both the Map<String, Catalog> catalogs and the CatalogStore at
> the same time can also avoid conflicts
>
>
> For lazy initialization:
> we can start from the Catalog itself and provide a dedicated Catalog
> implementation for lazy loading, such as ConfigurationCatalog
>
> public class ConfigurationCatalog implements Catalog {
> ConfigurationCatalog(Map<String, String> properties) {
> }
> }
>
> I have added this design to the FLIP.
>
>
> Best,
> Feng
>
> On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <zj...@gmail.com> wrote:
>
> > Hi Feng
> >
> > Thanks for your answer.
> >
> > I have no questions about the functionality of `CatalogStore`, but the
> > different operations of multiple `registerCatalog` or `storeCatalog` in
> > `CatalogManager`. The implementation in Trino is also the same, the
> > `CatalogManager` in trino only has one `createCatalog`, which will save
> the
> > catalog to memory and `CatalogStore`.
> >
> > I think we don't need to add another `registerCatalog` or `addCatalog` to
> > save a catalog in `Map<String, Catalog> catalogs` or `CatalogStore
> > catalogStore`. As you mentioned before, the `Map<String, Catalog>
> > catalogs` is a cache in `CatalogManager`. How about saving the catalog in
> > `Map<String, Catalog> catalogs` and `CatalogStore catalogStore` together
> > when it is registered or added in `CatalogManager`?
> >
> > Best,
> > Shammon FY
> >
> >
> > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <ji...@gmail.com> wrote:
> >
> > > Thank you for your reply. I am very sorry for the misunderstanding
> caused
> > > by my deviation from the original discussion.
> > >
> > > @Shammon
> > > > I found there is a pre-discussion [1] for this FLIP
> > > Yes, there was indeed such a discussion before. However, in designing
> > the
> > > whole solution, I found that the logic of CatalogManager itself doesn't
> > > need to change much. *We cannot only persist Catalog instances
> > themselves*,
> > > so exposing only registerCatalog(String catalogName, Catalog catalog)
> > might
> > > not be enough to save Catalogs, because in the end we still need to
> save
> > > the configurations corresponding to the Catalog instances. Therefore,
> I
> > > decided to introduce the CatalogStore interface for configuration
> > > persistence. Regarding this part, I also took inspiration from Trino's
> > > implementation[1].
> > >
> > >
> > > @yuxia
> > >
> > > > 1: The mechanism of handling catalog with the same name looks a
> little
> > of
> > > complex to me.
> > > Thank you for the suggestion. I will provide a detailed description and
> > > code examples for this part, and add it to the FLIP.
> > >
> > > > 2: TBH, the method name `addCatalog` still looks confused to me.
> IIUC,
> > > this method is for storing catalog to CatalogStore, how about renaming
> it
> > > to `storeCatalog`? It's very personal opinion, you can decide to take
> it
> > or
> > > not by your self.
> > > StoreCatalog looks more intuitive to me. I don't see any problem with
> it.
> > >
> > > > 3.For CREATE CATALOG statement, which method will be called?
> > > `registerCatalog` or `addCatalog`? I'm wondering whether user can add a
> > > catalog to store with SQL stement.
> > > For CREATE CATALOG, my original design was to add it directly to the
> > > CatalogStore, but this would disrupt the existing logic. Therefore, I
> > think
> > > we can do both: save the configuration to the CatalogStore and
> initialize
> > > the Catalog instance at the same time
> > >
> > > > 3. Is it really neccessary to provide a default implmentation for
> > > interface `CatalogStoreFactory`?
> > > I think it is necessary, otherwise we would need to introduce an
> > additional
> > > Map to store the configuration for lazy loading.
> > >
> > > > 4: About asynchronous registration for catalog.
> > > I don't think registerCatalog(String catalogName, Catalog catalog) can
> be
> > > made into an asynchronous interface because Catalog is already an
> > instance.
> > >
> > > > It looks more like lazy initialization for catalog than asynchronous
> > > registration, right?
> > > Indeed, my description was inaccurate. It should be lazy registration
> > > instead of asynchronous registration. I have already updated the title
> of
> > > the FLIP.
> > >
> > >
> > >
> > >
> > > [1].
> > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > >
> > >
> > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zj...@gmail.com> wrote:
> > >
> > > > Hi Feng
> > > >
> > > > I think if there is a `registerCatalog` method in `CatalogManager`,
> it
> > > will
> > > > confuse users whether a method named `addCatalog` or `storeCatalog`
> is
> > > > added.
> > > >
> > > > And as you mentioned, the memory catalog is a `cache`, I think the
> > > concept
> > > > of `cache` should not be exposed to users.
> > > >
> > > > I found there is a pre-discussion [1] for this FLIP. Please correct
> me
> > if
> > > > I'm wrong, IIUC, the conclusion of that discussion is to use
> > > > `CatalogManager` as an interface and implement it for different
> stores
> > > such
> > > > as memory, file and external system.
> > > >
> > > > I think there is a gap between the current FLIP design and that
> > > conclusion.
> > > > What about the proposal of the discussion in thread [1] ?
> > > >
> > > >
> > > > [1] https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > >
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > >
> > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > > wrote:
> > > >
> > > > > Thanks Feng for driving this FLIP.
> > > > > I have few comments:
> > > > > 1: The mechanism of handling catalog with the same name looks a
> > little
> > > of
> > > > > complex to me. I think it'll be better to explain it in the java
> doc
> > of
> > > > > these methods and give a brief example in this FLIP.
> > > > >
> > > > > 2: TBH, the method name `addCatalog` still looks confused to me.
> > IIUC,
> > > > > this method is for storing catalog to CatalogStore, how about
> > renaming
> > > it
> > > > > to `storeCatalog`? It's very personal opinion, you can decide to
> take
> > > it
> > > > or
> > > > > not by your self.
> > > > >
> > > > > 3: For CREATE CATALOG statement, which method will be called?
> > > > > `registerCatalog` or `addCatalog`? I'm wondering whether user can
> > add a
> > > > > catalog to store with SQL stement.
> > > > >
> > > > >
> > > > > 3: Is it really neccessary to provide a default implmentation for
> > > > > interface `CatalogStoreFactory`?
> > > > >
> > > > >
> > > > > 4: About asynchronous registration for catalog.
> > > > >
> > > > > > When creating a catalog with CREATE CATALOG, the asynchronous
> > > > > registration method is used by default.
> > > > > If asynchronous registration is the default behavior, it there any
> > way
> > > > > that user can switch to synchronous registration just like before?
> > > > > Will both method `addCatalog` and `registerCatalog` be asynchronous
> > > > > registration?
> > > > >
> > > > > IIUC, in asynchronous registration, it may well that CREATE CATALOG
> > > > > executes successfully, but then the following CREATE TABLE
> statement
> > > will
> > > > > fail for the catalog fail to open.
> > > > > I think it's a break change which should be highlighted in this
> FLIP,
> > > may
> > > > > be in compatibility part.
> > > > >
> > > > >
> > > > > BTW, by saying asynchronous registration, I would like to expect
> > there
> > > > > will be an executor to open or register catalog in the background,
> > but
> > > > from
> > > > > your previous comments,
> > > > > "the Catalog instance will be initialized if it has not been
> > > initialized
> > > > > yet. If the initialization process fails, these statements will not
> > be
> > > > > executed successfully."
> > > > > It looks more like lazy initialization for catalog than
> asynchronous
> > > > > registration, right?
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > > ----- 原始邮件 -----
> > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > configuration
> > > > > and asynchronous registration
> > > > >
> > > > > Hi everyone, Thank you all for your interest in this DISCUSS.
> > > > >
> > > > > @Shammon
> > > > > > How to handle a catalog with the same name that exists for both
> of
> > > > them?
> > > > >
> > > > > I believe this is a crucial point. Based on my current personal
> > > > > understanding, the Map<String, Catalog> catalogs will serve as a
> > cache
> > > > > for instantiated catalogs and have the highest priority.
> > > > >
> > > > > There are three methods that can affect the Map<String, Catalog>
> > > > catalogs:
> > > > >
> > > > > 1. registerCatalog(String catalogName, Catalog catalog)
> > > > >
> > > > > This method puts the catalog instance into the Map<String, Catalog>
> > > > > catalogs.
> > > > >
> > > > > 2. unregisterCatalog(String catalogName)
> > > > >
> > > > > This method removes the catalog instance corresponding to
> catalogName
> > > > > from the Map<String, Catalog> catalogs.
> > > > >
> > > > > 3. getCatalog(String catalogName)
> > > > >
> > > > > This method first retrieves the corresponding catalog instance from
> > > > > the Map<String, Catalog> catalogs. If the catalog does not exist,
> it
> > > > > retrieves the corresponding configuration from the CatalogStore,
> > > > > initializes it, and puts the initialized Catalog instance into the
> > > > > Map<String, Catalog> catalogs.
> > > > >
> > > > > The following two methods only modify the configuration in the
> > > > > CatalogStore:
> > > > >
> > > > > 1. addCatalog(String catalogName, Map<String, String> properties)
> > > > >
> > > > > This method saves the properties to the catalogStore and checks
> > > > > whether there is a catalogName with the same name.
> > > > >
> > > > > 2. removeCatalog(String catalogName)
> > > > > This method removes the specified configuration of the specified
> > > > > catalogName in the catalogStore.
> > > > >
> > > > > The following are possible conflict scenarios:
> > > > >
> > > > > 1. When the corresponding catalogName already exists in the
> > > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > > registerCatalog(String catalogName, Catalog catalog) method can
> > > > > succeed and be directly saved to the Map<String, Catalog> catalogs.
> > > > >
> > > > > 2. When the corresponding catalogName already exists in both the
> > > > > CatalogStore and the Map<String, Catalog>, the
> registerCatalog(String
> > > > > catalogName, Catalog catalog) method will fail.
> > > > >
> > > > > 3. When the corresponding catalogName already exists in the
> > > > > Map<String, Catalog>, the addCatalog(String catalogName,
> Map<String,
> > > > > String> properties) method can directly save the properties to the
> > > > > catalogStore, but the getCatalog(String catalogName) method will
> not
> > > > > use the new properties for initialization because the corresponding
> > > > > catalog instance already exists in catalogs and will be
> prioritized.
> > > > > Therefore, using the unregisterCatalog(String catalogName) method
> to
> > > > > remove the instance corresponding to the original catalogName is
> > > > > necessary.
> > > > >
> > > > >
> > > > >
> > > > > > I think it will confuse users that `registerCatalog(String
> > > > > catalogName,Catalog catalog)` in the `Map<String, Catalog>
> catalogs`
> > > and
> > > > > `registerCatalog(String catalogName, Map<String, String>
> properties)
> > > > >
> > > > > This could potentially lead to confusion. I suggest changing the
> > > > > method name, perhaps to addCatalog(String catalogName, Map<String,
> > > > > String> properties), as previously mentioned
> > > > >
> > > > >
> > > > > @Hang
> > > > > > add `registerCatalog(String catalogName,Catalog catalog,
> > > > > boolean lazyInit)` method
> > > > >
> > > > > Since a catalog is already an instance, adding the "lazyInit"
> > > > > parameter to the registerCatalog(String catalogName, Catalog
> catalog)
> > > > > method may not necessarily result in lazy initialization.
> > > > >
> > > > > > Do we need to think about encryption
> > > > >
> > > > > I think encryption is necessary, but perhaps the encryption logic
> > > > > should be implemented in the CatalogStore when the data is actually
> > > > > saved. For instance, the FileCatalogStore could encrypt the data
> when
> > > > > it is saved to a file, while the MemoryCatalogStore does not
> require
> > > > > encryption.
> > > > >
> > > > > > Do we really need the `MemoryCatalogStore`?
> > > > >
> > > > > I think it is necessary to have a MemoryCatalogStore as the default
> > > > > implementation, which saves Catalog configurations in memory.
> > > > > Otherwise, if we want to implement asynchronous loading of Catalog,
> > we
> > > > > would need to introduce additional cache.
> > > > >
> > > > >
> > > > > @Xianxun
> > > > >
> > > > > > What if asynchronous registration failed?
> > > > >
> > > > > This is also a critical concern. When executing DDL, DQL, or DML
> > > > > statements that reference a specific Catalog, the Catalog instance
> > > > > will be initialized if it has not been initialized yet. If the
> > > > > initialization process fails, these statements will not be executed
> > > > > successfully.
> > > > >
> > > > > > nt. The Map<String, Catalog> catalogs and CatalogStore
> catalogstore
> > > in
> > > > > the CatalogManager all have the same catalog name, but correponding
> > to
> > > > > different catalog instances.
> > > > >
> > > > > This issue can be resolved by referring to the previous responses.
> > The
> > > > > key principle is that the Catalog that has been most recently used
> > > > > will be given priority for subsequent use.
> > > > >
> > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <
> yesorno828423@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > Thanks for driving this Flip, I do believe this Flip could be
> > helpful
> > > > > for users. The firm I work for also manages a lot of catalogs, and
> > the
> > > > > submission of tasks becomes slow because of loading a number of
> > > catalogs.
> > > > > We obtain the catalogs in the user's Flink SQL through regular
> > > expression
> > > > > to avoid loading all catalogs to improve the speed of task
> > submission.
> > > > > After reading flip-295, I have some questions:
> > > > > >
> > > > > > 1. When creating a catalog with CREATE CATALOG, the asynchronous
> > > > > registration method is used by default. What if asynchronous
> > > registration
> > > > > failed? And how to implement asynchronous registration?
> > > > > >
> > > > > > 2. I also have the same question with Shammon’s first comment.
> The
> > > > > Map<String, Catalog> catalogs and CatalogStore catalogstore in the
> > > > > CatalogManager all have the same catalog name, but correponding to
> > > > > different catalog instances. For example: catalog1-> jdbcCatalog in
> > the
> > > > > catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would
> this
> > > > case
> > > > > happen? Maybe we should define `catalogs` and `catalogstore` more
> > > > clearly.
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Xianxun
> > > > > >
> > > > > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> > > > > > >
> > > > > > > MemoryCatalogStore
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
hi Shammon
Thank you for your response, and I completely agree with your point of view.
Initially, I may have over complicated the whole issue. First and foremost,
we need to consider the persistence of the Catalog's Configuration.
If we only need to provide persistence for Catalog Configuration, we can
add a toConfiguration method to the Catalog interface.
This method can convert a Catalog instance to a Map<String, String>
properties, and the default implementation will throw an exception.
public interface Catalog {
/**
* Returns a map containing the properties of the catalog object.
*
* @return a map containing the properties of the catalog object
* @throws UnsupportedOperationException if the implementing class does
not override
* the default implementation of this method
*/
default Map<String, String> toProperties() {
throw new UnsupportedOperationException("Please implement toProperties
for this catalog");
}
}
The specific process is as follows:
1. If the user has configured a CatalogStore, the toConfiguration() method
will be called when registering a Catalog instance with
registerCatalog(String catalogName, Catalog catalog). The Catalog instance
will be converted to a Map<String, String> properties and saved to the
CatalogStore. If some Catalog instances do not implement this method, an
exception will be thrown.
2. If the user does not use a CatalogStore, the toConfiguration() method
will not be called, ensuring consistency with the original process.
3. Saving both the Map<String, Catalog> catalogs and the CatalogStore at
the same time can also avoid conflicts
For lazy initialization:
we can start from the Catalog itself and provide a dedicated Catalog
implementation for lazy loading, such as ConfigurationCatalog
public class ConfigurationCatalog implements Catalog {
ConfigurationCatalog(Map<String, String> properties) {
}
}
I have added this design to the FLIP.
Best,
Feng
On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <zj...@gmail.com> wrote:
> Hi Feng
>
> Thanks for your answer.
>
> I have no questions about the functionality of `CatalogStore`, but the
> different operations of multiple `registerCatalog` or `storeCatalog` in
> `CatalogManager`. The implementation in Trino is also the same, the
> `CatalogManager` in trino only has one `createCatalog`, which will save the
> catalog to memory and `CatalogStore`.
>
> I think we don't need to add another `registerCatalog` or `addCatalog` to
> save a catalog in `Map<String, Catalog> catalogs` or `CatalogStore
> catalogStore`. As you mentioned before, the `Map<String, Catalog>
> catalogs` is a cache in `CatalogManager`. How about saving the catalog in
> `Map<String, Catalog> catalogs` and `CatalogStore catalogStore` together
> when it is registered or added in `CatalogManager`?
>
> Best,
> Shammon FY
>
>
> On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <ji...@gmail.com> wrote:
>
> > Thank you for your reply. I am very sorry for the misunderstanding caused
> > by my deviation from the original discussion.
> >
> > @Shammon
> > > I found there is a pre-discussion [1] for this FLIP
> > Yes, there was indeed such a discussion before. However, in designing
> the
> > whole solution, I found that the logic of CatalogManager itself doesn't
> > need to change much. *We cannot only persist Catalog instances
> themselves*,
> > so exposing only registerCatalog(String catalogName, Catalog catalog)
> might
> > not be enough to save Catalogs, because in the end we still need to save
> > the configurations corresponding to the Catalog instances. Therefore, I
> > decided to introduce the CatalogStore interface for configuration
> > persistence. Regarding this part, I also took inspiration from Trino's
> > implementation[1].
> >
> >
> > @yuxia
> >
> > > 1: The mechanism of handling catalog with the same name looks a little
> of
> > complex to me.
> > Thank you for the suggestion. I will provide a detailed description and
> > code examples for this part, and add it to the FLIP.
> >
> > > 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
> > this method is for storing catalog to CatalogStore, how about renaming it
> > to `storeCatalog`? It's very personal opinion, you can decide to take it
> or
> > not by your self.
> > StoreCatalog looks more intuitive to me. I don't see any problem with it.
> >
> > > 3.For CREATE CATALOG statement, which method will be called?
> > `registerCatalog` or `addCatalog`? I'm wondering whether user can add a
> > catalog to store with SQL stement.
> > For CREATE CATALOG, my original design was to add it directly to the
> > CatalogStore, but this would disrupt the existing logic. Therefore, I
> think
> > we can do both: save the configuration to the CatalogStore and initialize
> > the Catalog instance at the same time
> >
> > > 3. Is it really neccessary to provide a default implmentation for
> > interface `CatalogStoreFactory`?
> > I think it is necessary, otherwise we would need to introduce an
> additional
> > Map to store the configuration for lazy loading.
> >
> > > 4: About asynchronous registration for catalog.
> > I don't think registerCatalog(String catalogName, Catalog catalog) can be
> > made into an asynchronous interface because Catalog is already an
> instance.
> >
> > > It looks more like lazy initialization for catalog than asynchronous
> > registration, right?
> > Indeed, my description was inaccurate. It should be lazy registration
> > instead of asynchronous registration. I have already updated the title of
> > the FLIP.
> >
> >
> >
> >
> > [1].
> >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> >
> >
> > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zj...@gmail.com> wrote:
> >
> > > Hi Feng
> > >
> > > I think if there is a `registerCatalog` method in `CatalogManager`, it
> > will
> > > confuse users whether a method named `addCatalog` or `storeCatalog` is
> > > added.
> > >
> > > And as you mentioned, the memory catalog is a `cache`, I think the
> > concept
> > > of `cache` should not be exposed to users.
> > >
> > > I found there is a pre-discussion [1] for this FLIP. Please correct me
> if
> > > I'm wrong, IIUC, the conclusion of that discussion is to use
> > > `CatalogManager` as an interface and implement it for different stores
> > such
> > > as memory, file and external system.
> > >
> > > I think there is a gap between the current FLIP design and that
> > conclusion.
> > > What about the proposal of the discussion in thread [1] ?
> > >
> > >
> > > [1] https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > >
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > wrote:
> > >
> > > > Thanks Feng for driving this FLIP.
> > > > I have few comments:
> > > > 1: The mechanism of handling catalog with the same name looks a
> little
> > of
> > > > complex to me. I think it'll be better to explain it in the java doc
> of
> > > > these methods and give a brief example in this FLIP.
> > > >
> > > > 2: TBH, the method name `addCatalog` still looks confused to me.
> IIUC,
> > > > this method is for storing catalog to CatalogStore, how about
> renaming
> > it
> > > > to `storeCatalog`? It's very personal opinion, you can decide to take
> > it
> > > or
> > > > not by your self.
> > > >
> > > > 3: For CREATE CATALOG statement, which method will be called?
> > > > `registerCatalog` or `addCatalog`? I'm wondering whether user can
> add a
> > > > catalog to store with SQL stement.
> > > >
> > > >
> > > > 3: Is it really neccessary to provide a default implmentation for
> > > > interface `CatalogStoreFactory`?
> > > >
> > > >
> > > > 4: About asynchronous registration for catalog.
> > > >
> > > > > When creating a catalog with CREATE CATALOG, the asynchronous
> > > > registration method is used by default.
> > > > If asynchronous registration is the default behavior, it there any
> way
> > > > that user can switch to synchronous registration just like before?
> > > > Will both method `addCatalog` and `registerCatalog` be asynchronous
> > > > registration?
> > > >
> > > > IIUC, in asynchronous registration, it may well that CREATE CATALOG
> > > > executes successfully, but then the following CREATE TABLE statement
> > will
> > > > fail for the catalog fail to open.
> > > > I think it's a break change which should be highlighted in this FLIP,
> > may
> > > > be in compatibility part.
> > > >
> > > >
> > > > BTW, by saying asynchronous registration, I would like to expect
> there
> > > > will be an executor to open or register catalog in the background,
> but
> > > from
> > > > your previous comments,
> > > > "the Catalog instance will be initialized if it has not been
> > initialized
> > > > yet. If the initialization process fails, these statements will not
> be
> > > > executed successfully."
> > > > It looks more like lazy initialization for catalog than asynchronous
> > > > registration, right?
> > > >
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > ----- 原始邮件 -----
> > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > 收件人: "dev" <de...@flink.apache.org>
> > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > configuration
> > > > and asynchronous registration
> > > >
> > > > Hi everyone, Thank you all for your interest in this DISCUSS.
> > > >
> > > > @Shammon
> > > > > How to handle a catalog with the same name that exists for both of
> > > them?
> > > >
> > > > I believe this is a crucial point. Based on my current personal
> > > > understanding, the Map<String, Catalog> catalogs will serve as a
> cache
> > > > for instantiated catalogs and have the highest priority.
> > > >
> > > > There are three methods that can affect the Map<String, Catalog>
> > > catalogs:
> > > >
> > > > 1. registerCatalog(String catalogName, Catalog catalog)
> > > >
> > > > This method puts the catalog instance into the Map<String, Catalog>
> > > > catalogs.
> > > >
> > > > 2. unregisterCatalog(String catalogName)
> > > >
> > > > This method removes the catalog instance corresponding to catalogName
> > > > from the Map<String, Catalog> catalogs.
> > > >
> > > > 3. getCatalog(String catalogName)
> > > >
> > > > This method first retrieves the corresponding catalog instance from
> > > > the Map<String, Catalog> catalogs. If the catalog does not exist, it
> > > > retrieves the corresponding configuration from the CatalogStore,
> > > > initializes it, and puts the initialized Catalog instance into the
> > > > Map<String, Catalog> catalogs.
> > > >
> > > > The following two methods only modify the configuration in the
> > > > CatalogStore:
> > > >
> > > > 1. addCatalog(String catalogName, Map<String, String> properties)
> > > >
> > > > This method saves the properties to the catalogStore and checks
> > > > whether there is a catalogName with the same name.
> > > >
> > > > 2. removeCatalog(String catalogName)
> > > > This method removes the specified configuration of the specified
> > > > catalogName in the catalogStore.
> > > >
> > > > The following are possible conflict scenarios:
> > > >
> > > > 1. When the corresponding catalogName already exists in the
> > > > CatalogStore but not in the Map<String, Catalog>, the
> > > > registerCatalog(String catalogName, Catalog catalog) method can
> > > > succeed and be directly saved to the Map<String, Catalog> catalogs.
> > > >
> > > > 2. When the corresponding catalogName already exists in both the
> > > > CatalogStore and the Map<String, Catalog>, the registerCatalog(String
> > > > catalogName, Catalog catalog) method will fail.
> > > >
> > > > 3. When the corresponding catalogName already exists in the
> > > > Map<String, Catalog>, the addCatalog(String catalogName, Map<String,
> > > > String> properties) method can directly save the properties to the
> > > > catalogStore, but the getCatalog(String catalogName) method will not
> > > > use the new properties for initialization because the corresponding
> > > > catalog instance already exists in catalogs and will be prioritized.
> > > > Therefore, using the unregisterCatalog(String catalogName) method to
> > > > remove the instance corresponding to the original catalogName is
> > > > necessary.
> > > >
> > > >
> > > >
> > > > > I think it will confuse users that `registerCatalog(String
> > > > catalogName,Catalog catalog)` in the `Map<String, Catalog> catalogs`
> > and
> > > > `registerCatalog(String catalogName, Map<String, String> properties)
> > > >
> > > > This could potentially lead to confusion. I suggest changing the
> > > > method name, perhaps to addCatalog(String catalogName, Map<String,
> > > > String> properties), as previously mentioned
> > > >
> > > >
> > > > @Hang
> > > > > add `registerCatalog(String catalogName,Catalog catalog,
> > > > boolean lazyInit)` method
> > > >
> > > > Since a catalog is already an instance, adding the "lazyInit"
> > > > parameter to the registerCatalog(String catalogName, Catalog catalog)
> > > > method may not necessarily result in lazy initialization.
> > > >
> > > > > Do we need to think about encryption
> > > >
> > > > I think encryption is necessary, but perhaps the encryption logic
> > > > should be implemented in the CatalogStore when the data is actually
> > > > saved. For instance, the FileCatalogStore could encrypt the data when
> > > > it is saved to a file, while the MemoryCatalogStore does not require
> > > > encryption.
> > > >
> > > > > Do we really need the `MemoryCatalogStore`?
> > > >
> > > > I think it is necessary to have a MemoryCatalogStore as the default
> > > > implementation, which saves Catalog configurations in memory.
> > > > Otherwise, if we want to implement asynchronous loading of Catalog,
> we
> > > > would need to introduce additional cache.
> > > >
> > > >
> > > > @Xianxun
> > > >
> > > > > What if asynchronous registration failed?
> > > >
> > > > This is also a critical concern. When executing DDL, DQL, or DML
> > > > statements that reference a specific Catalog, the Catalog instance
> > > > will be initialized if it has not been initialized yet. If the
> > > > initialization process fails, these statements will not be executed
> > > > successfully.
> > > >
> > > > > nt. The Map<String, Catalog> catalogs and CatalogStore catalogstore
> > in
> > > > the CatalogManager all have the same catalog name, but correponding
> to
> > > > different catalog instances.
> > > >
> > > > This issue can be resolved by referring to the previous responses.
> The
> > > > key principle is that the Catalog that has been most recently used
> > > > will be given priority for subsequent use.
> > > >
> > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <ye...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi Feng,
> > > > >
> > > > > Thanks for driving this Flip, I do believe this Flip could be
> helpful
> > > > for users. The firm I work for also manages a lot of catalogs, and
> the
> > > > submission of tasks becomes slow because of loading a number of
> > catalogs.
> > > > We obtain the catalogs in the user's Flink SQL through regular
> > expression
> > > > to avoid loading all catalogs to improve the speed of task
> submission.
> > > > After reading flip-295, I have some questions:
> > > > >
> > > > > 1. When creating a catalog with CREATE CATALOG, the asynchronous
> > > > registration method is used by default. What if asynchronous
> > registration
> > > > failed? And how to implement asynchronous registration?
> > > > >
> > > > > 2. I also have the same question with Shammon’s first comment. The
> > > > Map<String, Catalog> catalogs and CatalogStore catalogstore in the
> > > > CatalogManager all have the same catalog name, but correponding to
> > > > different catalog instances. For example: catalog1-> jdbcCatalog in
> the
> > > > catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would this
> > > case
> > > > happen? Maybe we should define `catalogs` and `catalogstore` more
> > > clearly.
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Xianxun
> > > > >
> > > > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> > > > > >
> > > > > > MemoryCatalogStore
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Shammon FY <zj...@gmail.com>.
Hi Feng
Thanks for your answer.
I have no questions about the functionality of `CatalogStore`, but the
different operations of multiple `registerCatalog` or `storeCatalog` in
`CatalogManager`. The implementation in Trino is also the same, the
`CatalogManager` in trino only has one `createCatalog`, which will save the
catalog to memory and `CatalogStore`.
I think we don't need to add another `registerCatalog` or `addCatalog` to
save a catalog in `Map<String, Catalog> catalogs` or `CatalogStore
catalogStore`. As you mentioned before, the `Map<String, Catalog>
catalogs` is a cache in `CatalogManager`. How about saving the catalog in
`Map<String, Catalog> catalogs` and `CatalogStore catalogStore` together
when it is registered or added in `CatalogManager`?
Best,
Shammon FY
On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <ji...@gmail.com> wrote:
> Thank you for your reply. I am very sorry for the misunderstanding caused
> by my deviation from the original discussion.
>
> @Shammon
> > I found there is a pre-discussion [1] for this FLIP
> Yes, there was indeed such a discussion before. However, in designing the
> whole solution, I found that the logic of CatalogManager itself doesn't
> need to change much. *We cannot only persist Catalog instances themselves*,
> so exposing only registerCatalog(String catalogName, Catalog catalog) might
> not be enough to save Catalogs, because in the end we still need to save
> the configurations corresponding to the Catalog instances. Therefore, I
> decided to introduce the CatalogStore interface for configuration
> persistence. Regarding this part, I also took inspiration from Trino's
> implementation[1].
>
>
> @yuxia
>
> > 1: The mechanism of handling catalog with the same name looks a little of
> complex to me.
> Thank you for the suggestion. I will provide a detailed description and
> code examples for this part, and add it to the FLIP.
>
> > 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
> this method is for storing catalog to CatalogStore, how about renaming it
> to `storeCatalog`? It's very personal opinion, you can decide to take it or
> not by your self.
> StoreCatalog looks more intuitive to me. I don't see any problem with it.
>
> > 3.For CREATE CATALOG statement, which method will be called?
> `registerCatalog` or `addCatalog`? I'm wondering whether user can add a
> catalog to store with SQL stement.
> For CREATE CATALOG, my original design was to add it directly to the
> CatalogStore, but this would disrupt the existing logic. Therefore, I think
> we can do both: save the configuration to the CatalogStore and initialize
> the Catalog instance at the same time
>
> > 3. Is it really neccessary to provide a default implmentation for
> interface `CatalogStoreFactory`?
> I think it is necessary, otherwise we would need to introduce an additional
> Map to store the configuration for lazy loading.
>
> > 4: About asynchronous registration for catalog.
> I don't think registerCatalog(String catalogName, Catalog catalog) can be
> made into an asynchronous interface because Catalog is already an instance.
>
> > It looks more like lazy initialization for catalog than asynchronous
> registration, right?
> Indeed, my description was inaccurate. It should be lazy registration
> instead of asynchronous registration. I have already updated the title of
> the FLIP.
>
>
>
>
> [1].
>
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
>
>
> On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zj...@gmail.com> wrote:
>
> > Hi Feng
> >
> > I think if there is a `registerCatalog` method in `CatalogManager`, it
> will
> > confuse users whether a method named `addCatalog` or `storeCatalog` is
> > added.
> >
> > And as you mentioned, the memory catalog is a `cache`, I think the
> concept
> > of `cache` should not be exposed to users.
> >
> > I found there is a pre-discussion [1] for this FLIP. Please correct me if
> > I'm wrong, IIUC, the conclusion of that discussion is to use
> > `CatalogManager` as an interface and implement it for different stores
> such
> > as memory, file and external system.
> >
> > I think there is a gap between the current FLIP design and that
> conclusion.
> > What about the proposal of the discussion in thread [1] ?
> >
> >
> > [1] https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> >
> >
> > Best,
> > Shammon FY
> >
> >
> > On Tue, Apr 4, 2023 at 3:41 PM yuxia <lu...@alumni.sjtu.edu.cn>
> wrote:
> >
> > > Thanks Feng for driving this FLIP.
> > > I have few comments:
> > > 1: The mechanism of handling catalog with the same name looks a little
> of
> > > complex to me. I think it'll be better to explain it in the java doc of
> > > these methods and give a brief example in this FLIP.
> > >
> > > 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
> > > this method is for storing catalog to CatalogStore, how about renaming
> it
> > > to `storeCatalog`? It's very personal opinion, you can decide to take
> it
> > or
> > > not by your self.
> > >
> > > 3: For CREATE CATALOG statement, which method will be called?
> > > `registerCatalog` or `addCatalog`? I'm wondering whether user can add a
> > > catalog to store with SQL stement.
> > >
> > >
> > > 3: Is it really neccessary to provide a default implmentation for
> > > interface `CatalogStoreFactory`?
> > >
> > >
> > > 4: About asynchronous registration for catalog.
> > >
> > > > When creating a catalog with CREATE CATALOG, the asynchronous
> > > registration method is used by default.
> > > If asynchronous registration is the default behavior, it there any way
> > > that user can switch to synchronous registration just like before?
> > > Will both method `addCatalog` and `registerCatalog` be asynchronous
> > > registration?
> > >
> > > IIUC, in asynchronous registration, it may well that CREATE CATALOG
> > > executes successfully, but then the following CREATE TABLE statement
> will
> > > fail for the catalog fail to open.
> > > I think it's a break change which should be highlighted in this FLIP,
> may
> > > be in compatibility part.
> > >
> > >
> > > BTW, by saying asynchronous registration, I would like to expect there
> > > will be an executor to open or register catalog in the background, but
> > from
> > > your previous comments,
> > > "the Catalog instance will be initialized if it has not been
> initialized
> > > yet. If the initialization process fails, these statements will not be
> > > executed successfully."
> > > It looks more like lazy initialization for catalog than asynchronous
> > > registration, right?
> > >
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > ----- 原始邮件 -----
> > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > 收件人: "dev" <de...@flink.apache.org>
> > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> configuration
> > > and asynchronous registration
> > >
> > > Hi everyone, Thank you all for your interest in this DISCUSS.
> > >
> > > @Shammon
> > > > How to handle a catalog with the same name that exists for both of
> > them?
> > >
> > > I believe this is a crucial point. Based on my current personal
> > > understanding, the Map<String, Catalog> catalogs will serve as a cache
> > > for instantiated catalogs and have the highest priority.
> > >
> > > There are three methods that can affect the Map<String, Catalog>
> > catalogs:
> > >
> > > 1. registerCatalog(String catalogName, Catalog catalog)
> > >
> > > This method puts the catalog instance into the Map<String, Catalog>
> > > catalogs.
> > >
> > > 2. unregisterCatalog(String catalogName)
> > >
> > > This method removes the catalog instance corresponding to catalogName
> > > from the Map<String, Catalog> catalogs.
> > >
> > > 3. getCatalog(String catalogName)
> > >
> > > This method first retrieves the corresponding catalog instance from
> > > the Map<String, Catalog> catalogs. If the catalog does not exist, it
> > > retrieves the corresponding configuration from the CatalogStore,
> > > initializes it, and puts the initialized Catalog instance into the
> > > Map<String, Catalog> catalogs.
> > >
> > > The following two methods only modify the configuration in the
> > > CatalogStore:
> > >
> > > 1. addCatalog(String catalogName, Map<String, String> properties)
> > >
> > > This method saves the properties to the catalogStore and checks
> > > whether there is a catalogName with the same name.
> > >
> > > 2. removeCatalog(String catalogName)
> > > This method removes the specified configuration of the specified
> > > catalogName in the catalogStore.
> > >
> > > The following are possible conflict scenarios:
> > >
> > > 1. When the corresponding catalogName already exists in the
> > > CatalogStore but not in the Map<String, Catalog>, the
> > > registerCatalog(String catalogName, Catalog catalog) method can
> > > succeed and be directly saved to the Map<String, Catalog> catalogs.
> > >
> > > 2. When the corresponding catalogName already exists in both the
> > > CatalogStore and the Map<String, Catalog>, the registerCatalog(String
> > > catalogName, Catalog catalog) method will fail.
> > >
> > > 3. When the corresponding catalogName already exists in the
> > > Map<String, Catalog>, the addCatalog(String catalogName, Map<String,
> > > String> properties) method can directly save the properties to the
> > > catalogStore, but the getCatalog(String catalogName) method will not
> > > use the new properties for initialization because the corresponding
> > > catalog instance already exists in catalogs and will be prioritized.
> > > Therefore, using the unregisterCatalog(String catalogName) method to
> > > remove the instance corresponding to the original catalogName is
> > > necessary.
> > >
> > >
> > >
> > > > I think it will confuse users that `registerCatalog(String
> > > catalogName,Catalog catalog)` in the `Map<String, Catalog> catalogs`
> and
> > > `registerCatalog(String catalogName, Map<String, String> properties)
> > >
> > > This could potentially lead to confusion. I suggest changing the
> > > method name, perhaps to addCatalog(String catalogName, Map<String,
> > > String> properties), as previously mentioned
> > >
> > >
> > > @Hang
> > > > add `registerCatalog(String catalogName,Catalog catalog,
> > > boolean lazyInit)` method
> > >
> > > Since a catalog is already an instance, adding the "lazyInit"
> > > parameter to the registerCatalog(String catalogName, Catalog catalog)
> > > method may not necessarily result in lazy initialization.
> > >
> > > > Do we need to think about encryption
> > >
> > > I think encryption is necessary, but perhaps the encryption logic
> > > should be implemented in the CatalogStore when the data is actually
> > > saved. For instance, the FileCatalogStore could encrypt the data when
> > > it is saved to a file, while the MemoryCatalogStore does not require
> > > encryption.
> > >
> > > > Do we really need the `MemoryCatalogStore`?
> > >
> > > I think it is necessary to have a MemoryCatalogStore as the default
> > > implementation, which saves Catalog configurations in memory.
> > > Otherwise, if we want to implement asynchronous loading of Catalog, we
> > > would need to introduce additional cache.
> > >
> > >
> > > @Xianxun
> > >
> > > > What if asynchronous registration failed?
> > >
> > > This is also a critical concern. When executing DDL, DQL, or DML
> > > statements that reference a specific Catalog, the Catalog instance
> > > will be initialized if it has not been initialized yet. If the
> > > initialization process fails, these statements will not be executed
> > > successfully.
> > >
> > > > nt. The Map<String, Catalog> catalogs and CatalogStore catalogstore
> in
> > > the CatalogManager all have the same catalog name, but correponding to
> > > different catalog instances.
> > >
> > > This issue can be resolved by referring to the previous responses. The
> > > key principle is that the Catalog that has been most recently used
> > > will be given priority for subsequent use.
> > >
> > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <ye...@gmail.com>
> > > wrote:
> > > >
> > > > Hi Feng,
> > > >
> > > > Thanks for driving this Flip, I do believe this Flip could be helpful
> > > for users. The firm I work for also manages a lot of catalogs, and the
> > > submission of tasks becomes slow because of loading a number of
> catalogs.
> > > We obtain the catalogs in the user's Flink SQL through regular
> expression
> > > to avoid loading all catalogs to improve the speed of task submission.
> > > After reading flip-295, I have some questions:
> > > >
> > > > 1. When creating a catalog with CREATE CATALOG, the asynchronous
> > > registration method is used by default. What if asynchronous
> registration
> > > failed? And how to implement asynchronous registration?
> > > >
> > > > 2. I also have the same question with Shammon’s first comment. The
> > > Map<String, Catalog> catalogs and CatalogStore catalogstore in the
> > > CatalogManager all have the same catalog name, but correponding to
> > > different catalog instances. For example: catalog1-> jdbcCatalog in the
> > > catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would this
> > case
> > > happen? Maybe we should define `catalogs` and `catalogstore` more
> > clearly.
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Xianxun
> > > >
> > > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> > > > >
> > > > > MemoryCatalogStore
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Thank you for your reply. I am very sorry for the misunderstanding caused
by my deviation from the original discussion.
@Shammon
> I found there is a pre-discussion [1] for this FLIP
Yes, there was indeed such a discussion before. However, in designing the
whole solution, I found that the logic of CatalogManager itself doesn't
need to change much. *We cannot only persist Catalog instances themselves*,
so exposing only registerCatalog(String catalogName, Catalog catalog) might
not be enough to save Catalogs, because in the end we still need to save
the configurations corresponding to the Catalog instances. Therefore, I
decided to introduce the CatalogStore interface for configuration
persistence. Regarding this part, I also took inspiration from Trino's
implementation[1].
@yuxia
> 1: The mechanism of handling catalog with the same name looks a little of
complex to me.
Thank you for the suggestion. I will provide a detailed description and
code examples for this part, and add it to the FLIP.
> 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
this method is for storing catalog to CatalogStore, how about renaming it
to `storeCatalog`? It's very personal opinion, you can decide to take it or
not by your self.
StoreCatalog looks more intuitive to me. I don't see any problem with it.
> 3.For CREATE CATALOG statement, which method will be called?
`registerCatalog` or `addCatalog`? I'm wondering whether user can add a
catalog to store with SQL stement.
For CREATE CATALOG, my original design was to add it directly to the
CatalogStore, but this would disrupt the existing logic. Therefore, I think
we can do both: save the configuration to the CatalogStore and initialize
the Catalog instance at the same time
> 3. Is it really neccessary to provide a default implmentation for
interface `CatalogStoreFactory`?
I think it is necessary, otherwise we would need to introduce an additional
Map to store the configuration for lazy loading.
> 4: About asynchronous registration for catalog.
I don't think registerCatalog(String catalogName, Catalog catalog) can be
made into an asynchronous interface because Catalog is already an instance.
> It looks more like lazy initialization for catalog than asynchronous
registration, right?
Indeed, my description was inaccurate. It should be lazy registration
instead of asynchronous registration. I have already updated the title of
the FLIP.
[1].
https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <zj...@gmail.com> wrote:
> Hi Feng
>
> I think if there is a `registerCatalog` method in `CatalogManager`, it will
> confuse users whether a method named `addCatalog` or `storeCatalog` is
> added.
>
> And as you mentioned, the memory catalog is a `cache`, I think the concept
> of `cache` should not be exposed to users.
>
> I found there is a pre-discussion [1] for this FLIP. Please correct me if
> I'm wrong, IIUC, the conclusion of that discussion is to use
> `CatalogManager` as an interface and implement it for different stores such
> as memory, file and external system.
>
> I think there is a gap between the current FLIP design and that conclusion.
> What about the proposal of the discussion in thread [1] ?
>
>
> [1] https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
>
>
> Best,
> Shammon FY
>
>
> On Tue, Apr 4, 2023 at 3:41 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> > Thanks Feng for driving this FLIP.
> > I have few comments:
> > 1: The mechanism of handling catalog with the same name looks a little of
> > complex to me. I think it'll be better to explain it in the java doc of
> > these methods and give a brief example in this FLIP.
> >
> > 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
> > this method is for storing catalog to CatalogStore, how about renaming it
> > to `storeCatalog`? It's very personal opinion, you can decide to take it
> or
> > not by your self.
> >
> > 3: For CREATE CATALOG statement, which method will be called?
> > `registerCatalog` or `addCatalog`? I'm wondering whether user can add a
> > catalog to store with SQL stement.
> >
> >
> > 3: Is it really neccessary to provide a default implmentation for
> > interface `CatalogStoreFactory`?
> >
> >
> > 4: About asynchronous registration for catalog.
> >
> > > When creating a catalog with CREATE CATALOG, the asynchronous
> > registration method is used by default.
> > If asynchronous registration is the default behavior, it there any way
> > that user can switch to synchronous registration just like before?
> > Will both method `addCatalog` and `registerCatalog` be asynchronous
> > registration?
> >
> > IIUC, in asynchronous registration, it may well that CREATE CATALOG
> > executes successfully, but then the following CREATE TABLE statement will
> > fail for the catalog fail to open.
> > I think it's a break change which should be highlighted in this FLIP, may
> > be in compatibility part.
> >
> >
> > BTW, by saying asynchronous registration, I would like to expect there
> > will be an executor to open or register catalog in the background, but
> from
> > your previous comments,
> > "the Catalog instance will be initialized if it has not been initialized
> > yet. If the initialization process fails, these statements will not be
> > executed successfully."
> > It looks more like lazy initialization for catalog than asynchronous
> > registration, right?
> >
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Feng Jin" <ji...@gmail.com>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration
> > and asynchronous registration
> >
> > Hi everyone, Thank you all for your interest in this DISCUSS.
> >
> > @Shammon
> > > How to handle a catalog with the same name that exists for both of
> them?
> >
> > I believe this is a crucial point. Based on my current personal
> > understanding, the Map<String, Catalog> catalogs will serve as a cache
> > for instantiated catalogs and have the highest priority.
> >
> > There are three methods that can affect the Map<String, Catalog>
> catalogs:
> >
> > 1. registerCatalog(String catalogName, Catalog catalog)
> >
> > This method puts the catalog instance into the Map<String, Catalog>
> > catalogs.
> >
> > 2. unregisterCatalog(String catalogName)
> >
> > This method removes the catalog instance corresponding to catalogName
> > from the Map<String, Catalog> catalogs.
> >
> > 3. getCatalog(String catalogName)
> >
> > This method first retrieves the corresponding catalog instance from
> > the Map<String, Catalog> catalogs. If the catalog does not exist, it
> > retrieves the corresponding configuration from the CatalogStore,
> > initializes it, and puts the initialized Catalog instance into the
> > Map<String, Catalog> catalogs.
> >
> > The following two methods only modify the configuration in the
> > CatalogStore:
> >
> > 1. addCatalog(String catalogName, Map<String, String> properties)
> >
> > This method saves the properties to the catalogStore and checks
> > whether there is a catalogName with the same name.
> >
> > 2. removeCatalog(String catalogName)
> > This method removes the specified configuration of the specified
> > catalogName in the catalogStore.
> >
> > The following are possible conflict scenarios:
> >
> > 1. When the corresponding catalogName already exists in the
> > CatalogStore but not in the Map<String, Catalog>, the
> > registerCatalog(String catalogName, Catalog catalog) method can
> > succeed and be directly saved to the Map<String, Catalog> catalogs.
> >
> > 2. When the corresponding catalogName already exists in both the
> > CatalogStore and the Map<String, Catalog>, the registerCatalog(String
> > catalogName, Catalog catalog) method will fail.
> >
> > 3. When the corresponding catalogName already exists in the
> > Map<String, Catalog>, the addCatalog(String catalogName, Map<String,
> > String> properties) method can directly save the properties to the
> > catalogStore, but the getCatalog(String catalogName) method will not
> > use the new properties for initialization because the corresponding
> > catalog instance already exists in catalogs and will be prioritized.
> > Therefore, using the unregisterCatalog(String catalogName) method to
> > remove the instance corresponding to the original catalogName is
> > necessary.
> >
> >
> >
> > > I think it will confuse users that `registerCatalog(String
> > catalogName,Catalog catalog)` in the `Map<String, Catalog> catalogs` and
> > `registerCatalog(String catalogName, Map<String, String> properties)
> >
> > This could potentially lead to confusion. I suggest changing the
> > method name, perhaps to addCatalog(String catalogName, Map<String,
> > String> properties), as previously mentioned
> >
> >
> > @Hang
> > > add `registerCatalog(String catalogName,Catalog catalog,
> > boolean lazyInit)` method
> >
> > Since a catalog is already an instance, adding the "lazyInit"
> > parameter to the registerCatalog(String catalogName, Catalog catalog)
> > method may not necessarily result in lazy initialization.
> >
> > > Do we need to think about encryption
> >
> > I think encryption is necessary, but perhaps the encryption logic
> > should be implemented in the CatalogStore when the data is actually
> > saved. For instance, the FileCatalogStore could encrypt the data when
> > it is saved to a file, while the MemoryCatalogStore does not require
> > encryption.
> >
> > > Do we really need the `MemoryCatalogStore`?
> >
> > I think it is necessary to have a MemoryCatalogStore as the default
> > implementation, which saves Catalog configurations in memory.
> > Otherwise, if we want to implement asynchronous loading of Catalog, we
> > would need to introduce additional cache.
> >
> >
> > @Xianxun
> >
> > > What if asynchronous registration failed?
> >
> > This is also a critical concern. When executing DDL, DQL, or DML
> > statements that reference a specific Catalog, the Catalog instance
> > will be initialized if it has not been initialized yet. If the
> > initialization process fails, these statements will not be executed
> > successfully.
> >
> > > nt. The Map<String, Catalog> catalogs and CatalogStore catalogstore in
> > the CatalogManager all have the same catalog name, but correponding to
> > different catalog instances.
> >
> > This issue can be resolved by referring to the previous responses. The
> > key principle is that the Catalog that has been most recently used
> > will be given priority for subsequent use.
> >
> > On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <ye...@gmail.com>
> > wrote:
> > >
> > > Hi Feng,
> > >
> > > Thanks for driving this Flip, I do believe this Flip could be helpful
> > for users. The firm I work for also manages a lot of catalogs, and the
> > submission of tasks becomes slow because of loading a number of catalogs.
> > We obtain the catalogs in the user's Flink SQL through regular expression
> > to avoid loading all catalogs to improve the speed of task submission.
> > After reading flip-295, I have some questions:
> > >
> > > 1. When creating a catalog with CREATE CATALOG, the asynchronous
> > registration method is used by default. What if asynchronous registration
> > failed? And how to implement asynchronous registration?
> > >
> > > 2. I also have the same question with Shammon’s first comment. The
> > Map<String, Catalog> catalogs and CatalogStore catalogstore in the
> > CatalogManager all have the same catalog name, but correponding to
> > different catalog instances. For example: catalog1-> jdbcCatalog in the
> > catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would this
> case
> > happen? Maybe we should define `catalogs` and `catalogstore` more
> clearly.
> > >
> > >
> > > Best regards,
> > >
> > > Xianxun
> > >
> > > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> > > >
> > > > MemoryCatalogStore
> > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Shammon FY <zj...@gmail.com>.
Hi Feng
I think if there is a `registerCatalog` method in `CatalogManager`, it will
confuse users whether a method named `addCatalog` or `storeCatalog` is
added.
And as you mentioned, the memory catalog is a `cache`, I think the concept
of `cache` should not be exposed to users.
I found there is a pre-discussion [1] for this FLIP. Please correct me if
I'm wrong, IIUC, the conclusion of that discussion is to use
`CatalogManager` as an interface and implement it for different stores such
as memory, file and external system.
I think there is a gap between the current FLIP design and that conclusion.
What about the proposal of the discussion in thread [1] ?
[1] https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
Best,
Shammon FY
On Tue, Apr 4, 2023 at 3:41 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Thanks Feng for driving this FLIP.
> I have few comments:
> 1: The mechanism of handling catalog with the same name looks a little of
> complex to me. I think it'll be better to explain it in the java doc of
> these methods and give a brief example in this FLIP.
>
> 2: TBH, the method name `addCatalog` still looks confused to me. IIUC,
> this method is for storing catalog to CatalogStore, how about renaming it
> to `storeCatalog`? It's very personal opinion, you can decide to take it or
> not by your self.
>
> 3: For CREATE CATALOG statement, which method will be called?
> `registerCatalog` or `addCatalog`? I'm wondering whether user can add a
> catalog to store with SQL stement.
>
>
> 3: Is it really neccessary to provide a default implmentation for
> interface `CatalogStoreFactory`?
>
>
> 4: About asynchronous registration for catalog.
>
> > When creating a catalog with CREATE CATALOG, the asynchronous
> registration method is used by default.
> If asynchronous registration is the default behavior, it there any way
> that user can switch to synchronous registration just like before?
> Will both method `addCatalog` and `registerCatalog` be asynchronous
> registration?
>
> IIUC, in asynchronous registration, it may well that CREATE CATALOG
> executes successfully, but then the following CREATE TABLE statement will
> fail for the catalog fail to open.
> I think it's a break change which should be highlighted in this FLIP, may
> be in compatibility part.
>
>
> BTW, by saying asynchronous registration, I would like to expect there
> will be an executor to open or register catalog in the background, but from
> your previous comments,
> "the Catalog instance will be initialized if it has not been initialized
> yet. If the initialization process fails, these statements will not be
> executed successfully."
> It looks more like lazy initialization for catalog than asynchronous
> registration, right?
>
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Feng Jin" <ji...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration
> and asynchronous registration
>
> Hi everyone, Thank you all for your interest in this DISCUSS.
>
> @Shammon
> > How to handle a catalog with the same name that exists for both of them?
>
> I believe this is a crucial point. Based on my current personal
> understanding, the Map<String, Catalog> catalogs will serve as a cache
> for instantiated catalogs and have the highest priority.
>
> There are three methods that can affect the Map<String, Catalog> catalogs:
>
> 1. registerCatalog(String catalogName, Catalog catalog)
>
> This method puts the catalog instance into the Map<String, Catalog>
> catalogs.
>
> 2. unregisterCatalog(String catalogName)
>
> This method removes the catalog instance corresponding to catalogName
> from the Map<String, Catalog> catalogs.
>
> 3. getCatalog(String catalogName)
>
> This method first retrieves the corresponding catalog instance from
> the Map<String, Catalog> catalogs. If the catalog does not exist, it
> retrieves the corresponding configuration from the CatalogStore,
> initializes it, and puts the initialized Catalog instance into the
> Map<String, Catalog> catalogs.
>
> The following two methods only modify the configuration in the
> CatalogStore:
>
> 1. addCatalog(String catalogName, Map<String, String> properties)
>
> This method saves the properties to the catalogStore and checks
> whether there is a catalogName with the same name.
>
> 2. removeCatalog(String catalogName)
> This method removes the specified configuration of the specified
> catalogName in the catalogStore.
>
> The following are possible conflict scenarios:
>
> 1. When the corresponding catalogName already exists in the
> CatalogStore but not in the Map<String, Catalog>, the
> registerCatalog(String catalogName, Catalog catalog) method can
> succeed and be directly saved to the Map<String, Catalog> catalogs.
>
> 2. When the corresponding catalogName already exists in both the
> CatalogStore and the Map<String, Catalog>, the registerCatalog(String
> catalogName, Catalog catalog) method will fail.
>
> 3. When the corresponding catalogName already exists in the
> Map<String, Catalog>, the addCatalog(String catalogName, Map<String,
> String> properties) method can directly save the properties to the
> catalogStore, but the getCatalog(String catalogName) method will not
> use the new properties for initialization because the corresponding
> catalog instance already exists in catalogs and will be prioritized.
> Therefore, using the unregisterCatalog(String catalogName) method to
> remove the instance corresponding to the original catalogName is
> necessary.
>
>
>
> > I think it will confuse users that `registerCatalog(String
> catalogName,Catalog catalog)` in the `Map<String, Catalog> catalogs` and
> `registerCatalog(String catalogName, Map<String, String> properties)
>
> This could potentially lead to confusion. I suggest changing the
> method name, perhaps to addCatalog(String catalogName, Map<String,
> String> properties), as previously mentioned
>
>
> @Hang
> > add `registerCatalog(String catalogName,Catalog catalog,
> boolean lazyInit)` method
>
> Since a catalog is already an instance, adding the "lazyInit"
> parameter to the registerCatalog(String catalogName, Catalog catalog)
> method may not necessarily result in lazy initialization.
>
> > Do we need to think about encryption
>
> I think encryption is necessary, but perhaps the encryption logic
> should be implemented in the CatalogStore when the data is actually
> saved. For instance, the FileCatalogStore could encrypt the data when
> it is saved to a file, while the MemoryCatalogStore does not require
> encryption.
>
> > Do we really need the `MemoryCatalogStore`?
>
> I think it is necessary to have a MemoryCatalogStore as the default
> implementation, which saves Catalog configurations in memory.
> Otherwise, if we want to implement asynchronous loading of Catalog, we
> would need to introduce additional cache.
>
>
> @Xianxun
>
> > What if asynchronous registration failed?
>
> This is also a critical concern. When executing DDL, DQL, or DML
> statements that reference a specific Catalog, the Catalog instance
> will be initialized if it has not been initialized yet. If the
> initialization process fails, these statements will not be executed
> successfully.
>
> > nt. The Map<String, Catalog> catalogs and CatalogStore catalogstore in
> the CatalogManager all have the same catalog name, but correponding to
> different catalog instances.
>
> This issue can be resolved by referring to the previous responses. The
> key principle is that the Catalog that has been most recently used
> will be given priority for subsequent use.
>
> On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <ye...@gmail.com>
> wrote:
> >
> > Hi Feng,
> >
> > Thanks for driving this Flip, I do believe this Flip could be helpful
> for users. The firm I work for also manages a lot of catalogs, and the
> submission of tasks becomes slow because of loading a number of catalogs.
> We obtain the catalogs in the user's Flink SQL through regular expression
> to avoid loading all catalogs to improve the speed of task submission.
> After reading flip-295, I have some questions:
> >
> > 1. When creating a catalog with CREATE CATALOG, the asynchronous
> registration method is used by default. What if asynchronous registration
> failed? And how to implement asynchronous registration?
> >
> > 2. I also have the same question with Shammon’s first comment. The
> Map<String, Catalog> catalogs and CatalogStore catalogstore in the
> CatalogManager all have the same catalog name, but correponding to
> different catalog instances. For example: catalog1-> jdbcCatalog in the
> catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would this case
> happen? Maybe we should define `catalogs` and `catalogstore` more clearly.
> >
> >
> > Best regards,
> >
> > Xianxun
> >
> > > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> > >
> > > MemoryCatalogStore
> >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Thanks Feng for driving this FLIP.
I have few comments:
1: The mechanism of handling catalog with the same name looks a little of complex to me. I think it'll be better to explain it in the java doc of these methods and give a brief example in this FLIP.
2: TBH, the method name `addCatalog` still looks confused to me. IIUC, this method is for storing catalog to CatalogStore, how about renaming it to `storeCatalog`? It's very personal opinion, you can decide to take it or not by your self.
3: For CREATE CATALOG statement, which method will be called? `registerCatalog` or `addCatalog`? I'm wondering whether user can add a catalog to store with SQL stement.
3: Is it really neccessary to provide a default implmentation for interface `CatalogStoreFactory`?
4: About asynchronous registration for catalog.
> When creating a catalog with CREATE CATALOG, the asynchronous registration method is used by default.
If asynchronous registration is the default behavior, it there any way that user can switch to synchronous registration just like before?
Will both method `addCatalog` and `registerCatalog` be asynchronous registration?
IIUC, in asynchronous registration, it may well that CREATE CATALOG executes successfully, but then the following CREATE TABLE statement will fail for the catalog fail to open.
I think it's a break change which should be highlighted in this FLIP, may be in compatibility part.
BTW, by saying asynchronous registration, I would like to expect there will be an executor to open or register catalog in the background, but from your previous comments,
"the Catalog instance will be initialized if it has not been initialized yet. If the initialization process fails, these statements will not be executed successfully."
It looks more like lazy initialization for catalog than asynchronous registration, right?
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Feng Jin" <ji...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Hi everyone, Thank you all for your interest in this DISCUSS.
@Shammon
> How to handle a catalog with the same name that exists for both of them?
I believe this is a crucial point. Based on my current personal
understanding, the Map<String, Catalog> catalogs will serve as a cache
for instantiated catalogs and have the highest priority.
There are three methods that can affect the Map<String, Catalog> catalogs:
1. registerCatalog(String catalogName, Catalog catalog)
This method puts the catalog instance into the Map<String, Catalog> catalogs.
2. unregisterCatalog(String catalogName)
This method removes the catalog instance corresponding to catalogName
from the Map<String, Catalog> catalogs.
3. getCatalog(String catalogName)
This method first retrieves the corresponding catalog instance from
the Map<String, Catalog> catalogs. If the catalog does not exist, it
retrieves the corresponding configuration from the CatalogStore,
initializes it, and puts the initialized Catalog instance into the
Map<String, Catalog> catalogs.
The following two methods only modify the configuration in the CatalogStore:
1. addCatalog(String catalogName, Map<String, String> properties)
This method saves the properties to the catalogStore and checks
whether there is a catalogName with the same name.
2. removeCatalog(String catalogName)
This method removes the specified configuration of the specified
catalogName in the catalogStore.
The following are possible conflict scenarios:
1. When the corresponding catalogName already exists in the
CatalogStore but not in the Map<String, Catalog>, the
registerCatalog(String catalogName, Catalog catalog) method can
succeed and be directly saved to the Map<String, Catalog> catalogs.
2. When the corresponding catalogName already exists in both the
CatalogStore and the Map<String, Catalog>, the registerCatalog(String
catalogName, Catalog catalog) method will fail.
3. When the corresponding catalogName already exists in the
Map<String, Catalog>, the addCatalog(String catalogName, Map<String,
String> properties) method can directly save the properties to the
catalogStore, but the getCatalog(String catalogName) method will not
use the new properties for initialization because the corresponding
catalog instance already exists in catalogs and will be prioritized.
Therefore, using the unregisterCatalog(String catalogName) method to
remove the instance corresponding to the original catalogName is
necessary.
> I think it will confuse users that `registerCatalog(String catalogName,Catalog catalog)` in the `Map<String, Catalog> catalogs` and `registerCatalog(String catalogName, Map<String, String> properties)
This could potentially lead to confusion. I suggest changing the
method name, perhaps to addCatalog(String catalogName, Map<String,
String> properties), as previously mentioned
@Hang
> add `registerCatalog(String catalogName,Catalog catalog,
boolean lazyInit)` method
Since a catalog is already an instance, adding the "lazyInit"
parameter to the registerCatalog(String catalogName, Catalog catalog)
method may not necessarily result in lazy initialization.
> Do we need to think about encryption
I think encryption is necessary, but perhaps the encryption logic
should be implemented in the CatalogStore when the data is actually
saved. For instance, the FileCatalogStore could encrypt the data when
it is saved to a file, while the MemoryCatalogStore does not require
encryption.
> Do we really need the `MemoryCatalogStore`?
I think it is necessary to have a MemoryCatalogStore as the default
implementation, which saves Catalog configurations in memory.
Otherwise, if we want to implement asynchronous loading of Catalog, we
would need to introduce additional cache.
@Xianxun
> What if asynchronous registration failed?
This is also a critical concern. When executing DDL, DQL, or DML
statements that reference a specific Catalog, the Catalog instance
will be initialized if it has not been initialized yet. If the
initialization process fails, these statements will not be executed
successfully.
> nt. The Map<String, Catalog> catalogs and CatalogStore catalogstore in the CatalogManager all have the same catalog name, but correponding to different catalog instances.
This issue can be resolved by referring to the previous responses. The
key principle is that the Catalog that has been most recently used
will be given priority for subsequent use.
On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <ye...@gmail.com> wrote:
>
> Hi Feng,
>
> Thanks for driving this Flip, I do believe this Flip could be helpful for users. The firm I work for also manages a lot of catalogs, and the submission of tasks becomes slow because of loading a number of catalogs. We obtain the catalogs in the user's Flink SQL through regular expression to avoid loading all catalogs to improve the speed of task submission. After reading flip-295, I have some questions:
>
> 1. When creating a catalog with CREATE CATALOG, the asynchronous registration method is used by default. What if asynchronous registration failed? And how to implement asynchronous registration?
>
> 2. I also have the same question with Shammon’s first comment. The Map<String, Catalog> catalogs and CatalogStore catalogstore in the CatalogManager all have the same catalog name, but correponding to different catalog instances. For example: catalog1-> jdbcCatalog in the catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would this case happen? Maybe we should define `catalogs` and `catalogstore` more clearly.
>
>
> Best regards,
>
> Xianxun
>
> > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> >
> > MemoryCatalogStore
>
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi everyone, Thank you all for your interest in this DISCUSS.
@Shammon
> How to handle a catalog with the same name that exists for both of them?
I believe this is a crucial point. Based on my current personal
understanding, the Map<String, Catalog> catalogs will serve as a cache
for instantiated catalogs and have the highest priority.
There are three methods that can affect the Map<String, Catalog> catalogs:
1. registerCatalog(String catalogName, Catalog catalog)
This method puts the catalog instance into the Map<String, Catalog> catalogs.
2. unregisterCatalog(String catalogName)
This method removes the catalog instance corresponding to catalogName
from the Map<String, Catalog> catalogs.
3. getCatalog(String catalogName)
This method first retrieves the corresponding catalog instance from
the Map<String, Catalog> catalogs. If the catalog does not exist, it
retrieves the corresponding configuration from the CatalogStore,
initializes it, and puts the initialized Catalog instance into the
Map<String, Catalog> catalogs.
The following two methods only modify the configuration in the CatalogStore:
1. addCatalog(String catalogName, Map<String, String> properties)
This method saves the properties to the catalogStore and checks
whether there is a catalogName with the same name.
2. removeCatalog(String catalogName)
This method removes the specified configuration of the specified
catalogName in the catalogStore.
The following are possible conflict scenarios:
1. When the corresponding catalogName already exists in the
CatalogStore but not in the Map<String, Catalog>, the
registerCatalog(String catalogName, Catalog catalog) method can
succeed and be directly saved to the Map<String, Catalog> catalogs.
2. When the corresponding catalogName already exists in both the
CatalogStore and the Map<String, Catalog>, the registerCatalog(String
catalogName, Catalog catalog) method will fail.
3. When the corresponding catalogName already exists in the
Map<String, Catalog>, the addCatalog(String catalogName, Map<String,
String> properties) method can directly save the properties to the
catalogStore, but the getCatalog(String catalogName) method will not
use the new properties for initialization because the corresponding
catalog instance already exists in catalogs and will be prioritized.
Therefore, using the unregisterCatalog(String catalogName) method to
remove the instance corresponding to the original catalogName is
necessary.
> I think it will confuse users that `registerCatalog(String catalogName,Catalog catalog)` in the `Map<String, Catalog> catalogs` and `registerCatalog(String catalogName, Map<String, String> properties)
This could potentially lead to confusion. I suggest changing the
method name, perhaps to addCatalog(String catalogName, Map<String,
String> properties), as previously mentioned
@Hang
> add `registerCatalog(String catalogName,Catalog catalog,
boolean lazyInit)` method
Since a catalog is already an instance, adding the "lazyInit"
parameter to the registerCatalog(String catalogName, Catalog catalog)
method may not necessarily result in lazy initialization.
> Do we need to think about encryption
I think encryption is necessary, but perhaps the encryption logic
should be implemented in the CatalogStore when the data is actually
saved. For instance, the FileCatalogStore could encrypt the data when
it is saved to a file, while the MemoryCatalogStore does not require
encryption.
> Do we really need the `MemoryCatalogStore`?
I think it is necessary to have a MemoryCatalogStore as the default
implementation, which saves Catalog configurations in memory.
Otherwise, if we want to implement asynchronous loading of Catalog, we
would need to introduce additional cache.
@Xianxun
> What if asynchronous registration failed?
This is also a critical concern. When executing DDL, DQL, or DML
statements that reference a specific Catalog, the Catalog instance
will be initialized if it has not been initialized yet. If the
initialization process fails, these statements will not be executed
successfully.
> nt. The Map<String, Catalog> catalogs and CatalogStore catalogstore in the CatalogManager all have the same catalog name, but correponding to different catalog instances.
This issue can be resolved by referring to the previous responses. The
key principle is that the Catalog that has been most recently used
will be given priority for subsequent use.
On Sun, Apr 2, 2023 at 10:58 PM Xianxun Ye <ye...@gmail.com> wrote:
>
> Hi Feng,
>
> Thanks for driving this Flip, I do believe this Flip could be helpful for users. The firm I work for also manages a lot of catalogs, and the submission of tasks becomes slow because of loading a number of catalogs. We obtain the catalogs in the user's Flink SQL through regular expression to avoid loading all catalogs to improve the speed of task submission. After reading flip-295, I have some questions:
>
> 1. When creating a catalog with CREATE CATALOG, the asynchronous registration method is used by default. What if asynchronous registration failed? And how to implement asynchronous registration?
>
> 2. I also have the same question with Shammon’s first comment. The Map<String, Catalog> catalogs and CatalogStore catalogstore in the CatalogManager all have the same catalog name, but correponding to different catalog instances. For example: catalog1-> jdbcCatalog in the catalogs, but catalog1 -> hiveCatalog in the catalogstore. Would this case happen? Maybe we should define `catalogs` and `catalogstore` more clearly.
>
>
> Best regards,
>
> Xianxun
>
> > 2023年3月31日 11:13，Hang Ruan <ru...@gmail.com> 写道：
> >
> > MemoryCatalogStore
>
>