[jira] [Commented] (FLINK-9172) Support external catalog factory that comes default with SQL-Client

["Rong Rong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601690#comment-16601690 ] 

Rong Rong commented on FLINK-9172:
----------------------------------

Hi folks. Sorry for not taking this task for so long. I think [~twalthr] had put int a very nice unification on external catalog by merging the change in [https://github.com/apache/flink/pull/6343] for FLLINK-9852.

As of now I think there are several things missing:
 1. how the external catalogs can be configured in SQL-client. I am thinking adding the following in sql-client yaml config:
{code}
#==============================================================================
# Data Catalogs
#==============================================================================

catalogs: [] # empty list
# A typical data catalog definition looks like: 
# - name: ...
#   catalog-type: [? extends ExternalCatalog.class] # such as HCatalog, AvroCatalog
#   datastore-type: [batch, streaming]
#   connection: [? extends ConnectorDescriptor]
{code}
since ExternalCatalogTable only requires (1) a concrete implementation class and (2) ConnectorDescriptor for instantiate the connection

2. how to use the ExternalCatalog
We definitely by default register all {{ExternalCatalog}} implementations into {{TableEnvironment}}. I tried with a POC and it can be easily put into use by simply invoking the external catalog tables with
{code}
SELECT cols FROM catalog_name.subschema_name.table_name
{code}
I haven't put too much thought in to the {{TableSink}} part, maybe this is already supported by 
{code}
INSERT INTO catalog_name.table_sink_name SELECT * FROM catalog_name.table_source_name
{code}

[~eronwright] what do you think about the config YAML change. anything else we can add? I was thinking about credential / security but they can probably be put in as part of the ConnectorDescriptor

> Support external catalog factory that comes default with SQL-Client
> -------------------------------------------------------------------
>
>                 Key: FLINK-9172
>                 URL: https://issues.apache.org/jira/browse/FLINK-9172
>             Project: Flink
>          Issue Type: Sub-task
>            Reporter: Rong Rong
>            Assignee: Rong Rong
>            Priority: Major
>
> It will be great to have SQL-Client to support some external catalogs out-of-the-box for SQL users to configure and utilize easily. I am currently think of having an external catalog factory that spins up both streaming and batch external catalog table sources and sinks. This could greatly unify and provide easy access for SQL users. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)