You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jane Chan (Jira)" <ji...@apache.org> on 2021/02/07 02:08:00 UTC
[jira] [Comment Edited] (FLINK-21045) Improve Usability of
Pluggable Modules
[ https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277867#comment-17277867 ]
Jane Chan edited comment on FLINK-21045 at 2/7/21, 2:07 AM:
------------------------------------------------------------
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is anything missed or not corrected, please let me know. For more information, please reach to the [discussion thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html]. This summary is also backed up on the [google doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
----
h2. API Change
h3. ModuleManager
We introduce two methods to support {{USE MODULES}} and {{SHOW FULL MODULES}}. Beyond that, the behavior of methods {{listModules}}, {{listFunctions}}, and {{getFunctionDefinition}} will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List<ModuleEntry> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded but not on the
* name list will become unused but still loaded with relative order unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(String... names) {
// list all loaded modules with status
}
/** A POJO to represent a module's name and status. */
static class ModuleEntry {
private final String name;
private final boolean used;
ModuleEntry(String name, boolean used) {
this.name = name;
this.used = used;
// getters
}
}
{code}
h3. TableEnvironment
We introduce two methods {{listFullModules()}} and {{useModules(String… names)}} to {{TableEnvironment}} interface. The impl will simply call {{moduleManager#listFullModules()}} and {{moduleManager#useModules(String… names)}}.
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides, there will be a long-term plan to migrate {{ModuleFactory}} to extends {{org.apache.flink.table.factories.Factory}}. This plan is currently beyond the scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
{code:java}
LOAD MODULE module_name [WITH ('prop' = 'myProp', ...)]{code}
Explanation
* Module name {{module_name}} is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
-------------------
Flink SQL> LOAD MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
-------------------
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the same relative positions.
{code:java}
UNLOAD MODULE module_name{code}
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
| ml | false |
-------------------
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
-------------------
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
{code:java}
USE MODULES x [,y, z, ...]{code}
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...] in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared module list contains a not loaded module.
* There is no guarantee on orders between unused modules.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name [unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
{code}
h3. SHOW [FULL] MODULES
Display the loaded module names in declared order.
{code:java}
SHOW [FULL] MODULES{code}
Explanation
* If `FULL` is specified, all loaded modules and their status will be listed.
* If `FULL` is omitted, only used modules will be listed.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> SHOW MODULES;
--------------
| modules |
--------------
| core |
--------------
{code}
h2. YAML Change
“type” is deprecated and removed from the configuration.
{code:yaml}
modules:
- name: core
- name: hive
hive-version: 2.2.1
{code}
Best,
Jane
was (Author: qingyue):
Hi everyone,
I did a summary since the discussion has achieved a consensus. If there is anything missed or not corrected, please let me know. For more information, please reach to the [discussion thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html]. This summary is also backed up on the [google doc|https://docs.google.com/document/d/111d9ZOdI0JDzTUlG7s2ki6_jygHkVctPjsjrNknwyCk/edit#heading=h.gsxtfw7wz6pl].
----
h2. API Change
h3. ModuleManager
We introduce two methods to support {{USE MODULES}} and {{SHOW FULL MODULES}}. Beyond that, the behavior of methods {{listModules}}, {{listFunctions}}, and {{getFunctionDefinition}} will return functions and definitions of used modules.
{code:java}
public class ModuleManager {
/**
* Get names and use status of all modules loaded.
*
* @return a list of pairs of name and status for all modules loaded.
*/
public List<ModuleEntry> listFullModules() {
// list all loaded modules with status
}
/**
* Enable modules with declared name order. Modules that have been loaded but not on the
* name list will become unused but still loaded with relative order unchanged.
*
* @param names module names to be used.
* @throws ValidationException when module names contain an unloaded name.
*/
public void useModules(String... names) {
// list all loaded modules with status
}
/** A POJO to represent a module's name and status. */
static class ModuleEntry {
private final String name;
private final boolean used;
ModuleEntry(String name, boolean used) {
this.name = name;
this.used = used;
// getters
}
}
{code}
h3. TableEnvironment
We introduce two methods {{listFullModules()}} and {{useModules(String… names)}} to {{TableEnvironment}} interface. The impl will simply call {{moduleManager#listFullModules()}} and {{moduleManager#useModules(String… names)}}.
h3. ModuleFactory
ModuleFactory will use the module name to perform factory discovery. Besides, there will be a long-term plan to migrate {{ModuleFactory}} to extends {{org.apache.flink.table.factories.Factory}}. This plan is currently beyond the scope of this improvement.
h2. SQL Syntax Change
h3. LOAD MODULE
Load a module with the given name and append it to the end of the module list.
{code:java}
LOAD MODULE module_name [WITH ('prop' = 'myProp', ...)]{code}
Explanation
* Module name {{module_name}} is a simple identifier instead of a string literal.
* Property key 'type' is deprecated and removed now, and factory discovery will use the module name.
* Throws org.apache.flink.table.api.ValidationException if module_name exists.
* The newly loaded module will be implicitly used by default.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
-------------------
Flink SQL> LOAD MODULE hive WITH ('hive-version' = '3.1.2');
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
-------------------
{code}
h3. UNLOAD MODULE
Unload a module by name from the module list, and other modules remain in the same relative positions.
{code:java}
UNLOAD MODULE module_name{code}
Explanation
* Throws org.apache.flink.table.api.ValidationException if module_name does not exist.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| hive | true |
| ml | false |
-------------------
Flink SQL> UNLOAD MODULE hive;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
-------------------
{code}
USE MODULES
Change resolution order for loaded modules and meanwhile enable them in use.
{code:java}
USE MODULES x [,y, z, ...]{code}
Explanation
* This behavior will change the resolution priority for modules x [,y, z,...] in the declared order, and meanwhile, enable them.
* The unmentioned modules will become 'inactive' but still loaded.
* Throws org.apache.flink.table.api.ValidationException once the declared module list contains a not loaded module.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> USE MODULE hive, ml;
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
Flink SQL> USE MODULES core, unexisted_some_module, ml;
[ERROR] Could not execute SQL statement [USE]. Reason:
org.apache.flink.table.api.ValidationException: A module with name [unexisted_some_module] does not exist.
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| hive | true |
| ml | true |
| core | false |
-------------------
{code}
h3. SHOW [FULL] MODULES
Display the loaded module names in declared order.
{code:java}
SHOW [FULL] MODULES{code}
Explanation
* If `FULL` is specified, all loaded modules and their status will be listed.
* If `FULL` is omitted, only used modules will be listed.
Example
{code:java}
Flink SQL> SHOW FULL MODULES;
-------------------
| modules | used |
-------------------
| core | true |
| ml | false |
| hive | false |
-------------------
Flink SQL> SHOW MODULES;
--------------
| modules |
--------------
| core |
--------------
{code}
h2. YAML Change
“type” is deprecated and removed from the configuration.
{code:yaml}
modules:
- name: core
- name: hive
hive-version: 2.2.1
{code}
Best,
Jane
> Improve Usability of Pluggable Modules
> --------------------------------------
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
> Affects Versions: 1.13.0
> Reporter: Nicholas Jiang
> Assignee: Jane Chan
> Priority: Major
> Fix For: 1.13.0
>
>
> This improvement aims to
> # Simplify the module discovery mapping by module name. This will encourage users to create singleton of module instances.
> # Support changing the resolution order of modules in a flexible manner. This will introduce two methods {{#useModules}} and {{#listFullModules}} in both {{ModuleManager}} and {{TableEnvironment}}.
> # Support SQL syntax upon {{LOAD/UNLOAD MODULE}}, {{USE MODULES}}, and {{SHOW [FULL] MODULES}} in both {{FlinkSqlParserImpl}} and {{SqlClient}}.
> # Update the documentation to keep users informed of this improvement.
> Please reach to the [discussion thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLINK-21045-Support-load-module-and-unload-module-SQL-syntax-td48398.html] for more details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)