You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "LadyForest (via GitHub)" <gi...@apache.org> on 2023/01/29 08:26:57 UTC
[GitHub] [flink] LadyForest commented on a diff in pull request #21752: [FLINK-30691][sql-client] Supports the gateway mode in the sql-client
LadyForest commented on code in PR #21752:
URL: https://github.com/apache/flink/pull/21752#discussion_r1089891027
##########
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/DefaultContextUtils.java:
##########
@@ -69,6 +69,15 @@ public static DefaultContext buildDefaultContext(CliOptions options) {
return new DefaultContext(configuration, dependencies);
}
+ public static DefaultContext buildDefaultContext(CliOptions.GatewayCliOptions options) {
Review Comment:
`options` not used?
##########
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliOptionsParser.java:
##########
@@ -285,35 +268,18 @@ public static CliOptions parseGatewayModeClient(String[] args) {
try {
DefaultParser parser = new DefaultParser();
CommandLine line = parser.parse(GATEWAY_MODE_CLIENT_OPTIONS, args, true);
- return new CliOptions(
+ return new CliOptions.GatewayCliOptions(
line.hasOption(CliOptionsParser.OPTION_HELP.getOpt()),
checkSessionId(line),
- null,
- null,
- checkUrls(line, CliOptionsParser.OPTION_JAR),
- checkUrls(line, CliOptionsParser.OPTION_LIBRARY),
+ checkUrl(line, CliOptionsParser.OPTION_INIT_FILE),
+ checkUrl(line, CliOptionsParser.OPTION_FILE),
line.getOptionValue(CliOptionsParser.OPTION_UPDATE.getOpt()),
line.getOptionValue(CliOptionsParser.OPTION_HISTORY.getOpt()),
- getPythonConfiguration(line));
- } catch (ParseException e) {
- throw new SqlClientException(e.getMessage());
- }
- }
-
- public static CliOptions parseGatewayModeGateway(String[] args) {
- try {
- DefaultParser parser = new DefaultParser();
- CommandLine line = parser.parse(GATEWAY_MODE_GATEWAY_OPTIONS, args, true);
- return new CliOptions(
- line.hasOption(CliOptionsParser.OPTION_HELP.getOpt()),
- null,
- null,
- null,
- checkUrls(line, CliOptionsParser.OPTION_JAR),
- checkUrls(line, CliOptionsParser.OPTION_LIBRARY),
- null,
- null,
- getPythonConfiguration(line));
+ line.hasOption(CliOptionsParser.OPTION_ENDPOINT_ADDRESS.getOpt())
+ ? NetUtils.parseHostPortAddress(
+ line.getOptionValue(
+ CliOptionsParser.OPTION_ENDPOINT_ADDRESS.getOpt()))
Review Comment:
Just curious about what a `null` endpoint means.
##########
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/SqlClient.java:
##########
@@ -56,38 +57,52 @@
* <p>- In embedded mode, the SQL CLI is tightly coupled with the executor in a common process. This
* allows for submitting jobs without having to start an additional component.
*
- * <p>- In future versions: In gateway mode, the SQL CLI client connects to the REST API of the
- * gateway and allows for managing queries via console.
- *
- * <p>For debugging in an IDE you can execute the main method of this class using: "--defaults
- * /path/to/sql-client-defaults.yaml --jar /path/to/target/flink-sql-client-*.jar"
- *
- * <p>Make sure that the FLINK_CONF_DIR environment variable is set.
+ * <p>- In gateway mode, the SQL CLI client connects to the REST API of the gateway and allows for
+ * managing queries via console.
*/
public class SqlClient {
private static final Logger LOG = LoggerFactory.getLogger(SqlClient.class);
- private final boolean isEmbedded;
+ private final boolean isGatewayMode;
private final CliOptions options;
private final Supplier<Terminal> terminalFactory;
public static final String MODE_EMBEDDED = "embedded";
public static final String MODE_GATEWAY = "gateway";
+ public static final String MODE_NONE = "";
Review Comment:
I think it's better to have all kinds of `MODE_XX` as `Enum` instead of `String`
##########
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/SqlClient.java:
##########
@@ -265,9 +289,13 @@ public void close() {
private static class EmbeddedShutdownThread extends Thread {
Review Comment:
If `embedded` and `gateway` shares the shutdown hook logic, we'd better rename `EmbeddedShutdownThread` to `ShutdownThread`
##########
flink-table/flink-sql-client/src/test/resources/cli/all-mode-help.out:
##########
@@ -0,0 +1,175 @@
+./sql-client [MODE] [OPTIONS]
Review Comment:
What I've got
```text
./bin/sql-client.sh -h
usage:
-f,--file <script file> Script file that should be
executed. In this mode, the
client will not open an
interactive terminal.
-h,--help Show the help message with
descriptions of all options.
-hist,--history <History file path> The file which you want to save
the command history into. If not
specified, we will auto-generate
one under your user's home
directory.
-i,--init <initialization file> Script file that used to init
the session context. If get
error in execution, the sql
client will exit. Notice it's
not allowed to add query or
insert into the init file.
-j,--jar <JAR file> A JAR file to be imported into
the session. The file might
contain user-defined classes
needed for the execution of
statements such as functions,
table sources, or sinks. Can be
used multiple times.
-l,--library <JAR directory> A JAR file directory with which
every new session is
initialized. The files might
contain user-defined classes
needed for the execution of
statements such as functions,
table sources, or sinks. Can be
used multiple times.
-pyarch,--pyArchives <arg> Add python archive files for
job. The archive files will be
extracted to the working
directory of python UDF worker.
For each archive file, a target
directory be specified. If the
target directory name is
specified, the archive file will
be extracted to a directory with
the specified name. Otherwise,
the archive file will be
extracted to a directory with
the same name of the archive
file. The files uploaded via
this option are accessible via
relative path. '#' could be used
as the separator of the archive
file path and the target
directory name. Comma (',')
could be used as the separator
to specify multiple archive
files. This option can be used
to upload the virtual
environment, the data files used
in Python UDF (e.g.,
--pyArchives
file:///tmp/py37.zip,file:///tmp
/data.zip#data --pyExecutable
py37.zip/py37/bin/python). The
data files could be accessed in
Python UDF, e.g.: f =
open('data/data.txt', 'r').
-pyclientexec,--pyClientExecutable <arg> The path of the Python
interpreter used to launch the
Python process when submitting
the Python jobs via "flink run"
or compiling the Java/Scala jobs
containing Python UDFs.
-pyexec,--pyExecutable <arg> Specify the path of the python
interpreter used to execute the
python UDF worker (e.g.:
--pyExecutable
/usr/local/bin/python3). The
python UDF worker depends on
Python 3.7+, Apache Beam
(version == 2.43.0), Pip
(version >= 20.3) and SetupTools
(version >= 37.0.0). Please
ensure that the specified
environment meets the above
requirements.
-pyfs,--pyFiles <pythonFiles> Attach custom files for job. The
standard resource file suffixes
such as .py/.egg/.zip/.whl or
directory are all supported.
These files will be added to the
PYTHONPATH of both the local
client and the remote python UDF
worker. Files suffixed with .zip
will be extracted and added to
PYTHONPATH. Comma (',') could be
used as the separator to specify
multiple files (e.g., --pyFiles
file:///tmp/myresource.zip,hdfs:
///$namenode_address/myresource2
.zip).
-pyreq,--pyRequirements <arg> Specify a requirements.txt file
which defines the third-party
dependencies. These dependencies
will be installed and added to
the PYTHONPATH of the python UDF
worker. A directory which
contains the installation
packages of these dependencies
could be specified optionally.
Use '#' as the separator if the
optional parameter exists (e.g.,
--pyRequirements
file:///tmp/requirements.txt#fil
e:///tmp/cached_dir).
-s,--session <session identifier> The identifier for a session.
'default' is the default
identifier.
-u,--update <SQL update statement> Deprecated Experimental (for
testing only!) feature:
Instructs the SQL Client to
immediately execute the given
update statement after starting
up. The process is shut down
after the statement has been
submitted to the cluster and
returns an appropriate return
code. Currently, this feature is
only supported for INSERT INTO
statements that declare the
target sink table.Please use
option -f to submit update
statement.
./sql-client [MODE] [OPTIONS]
The following options are available:
Mode "embedded" (default) submits Flink jobs from the local machine.
Syntax: [embedded] [OPTIONS]
"embedded" mode options:
-f,--file <script file> Script file that should be
executed. In this mode, the
client will not open an
interactive terminal.
-h,--help Show the help message with
descriptions of all options.
-hist,--history <History file path> The file which you want to save
the command history into. If not
specified, we will auto-generate
one under your user's home
directory.
-i,--init <initialization file> Script file that used to init
the session context. If get
error in execution, the sql
client will exit. Notice it's
not allowed to add query or
insert into the init file.
-j,--jar <JAR file> A JAR file to be imported into
the session. The file might
contain user-defined classes
needed for the execution of
statements such as functions,
table sources, or sinks. Can be
used multiple times.
-l,--library <JAR directory> A JAR file directory with which
every new session is
initialized. The files might
contain user-defined classes
needed for the execution of
statements such as functions,
table sources, or sinks. Can be
used multiple times.
-pyarch,--pyArchives <arg> Add python archive files for
job. The archive files will be
extracted to the working
directory of python UDF worker.
For each archive file, a target
directory be specified. If the
target directory name is
specified, the archive file will
be extracted to a directory with
the specified name. Otherwise,
the archive file will be
extracted to a directory with
the same name of the archive
file. The files uploaded via
this option are accessible via
relative path. '#' could be used
as the separator of the archive
file path and the target
directory name. Comma (',')
could be used as the separator
to specify multiple archive
files. This option can be used
to upload the virtual
environment, the data files used
in Python UDF (e.g.,
--pyArchives
file:///tmp/py37.zip,file:///tmp
/data.zip#data --pyExecutable
py37.zip/py37/bin/python). The
data files could be accessed in
Python UDF, e.g.: f =
open('data/data.txt', 'r').
-pyclientexec,--pyClientExecutable <arg> The path of the Python
interpreter used to launch the
Python process when submitting
the Python jobs via "flink run"
or compiling the Java/Scala jobs
containing Python UDFs.
-pyexec,--pyExecutable <arg> Specify the path of the python
interpreter used to execute the
python UDF worker (e.g.:
--pyExecutable
/usr/local/bin/python3). The
python UDF worker depends on
Python 3.7+, Apache Beam
(version == 2.43.0), Pip
(version >= 20.3) and SetupTools
(version >= 37.0.0). Please
ensure that the specified
environment meets the above
requirements.
-pyfs,--pyFiles <pythonFiles> Attach custom files for job. The
standard resource file suffixes
such as .py/.egg/.zip/.whl or
directory are all supported.
These files will be added to the
PYTHONPATH of both the local
client and the remote python UDF
worker. Files suffixed with .zip
will be extracted and added to
PYTHONPATH. Comma (',') could be
used as the separator to specify
multiple files (e.g., --pyFiles
file:///tmp/myresource.zip,hdfs:
///$namenode_address/myresource2
.zip).
-pyreq,--pyRequirements <arg> Specify a requirements.txt file
which defines the third-party
dependencies. These dependencies
will be installed and added to
the PYTHONPATH of the python UDF
worker. A directory which
contains the installation
packages of these dependencies
could be specified optionally.
Use '#' as the separator if the
optional parameter exists (e.g.,
--pyRequirements
file:///tmp/requirements.txt#fil
e:///tmp/cached_dir).
-s,--session <session identifier> The identifier for a session.
'default' is the default
identifier.
-u,--update <SQL update statement> Deprecated Experimental (for
testing only!) feature:
Instructs the SQL Client to
immediately execute the given
update statement after starting
up. The process is shut down
after the statement has been
submitted to the cluster and
returns an appropriate return
code. Currently, this feature is
only supported for INSERT INTO
statements that declare the
target sink table.Please use
option -f to submit update
statement.
Mode "gateway" mode connects to the SQL gateway for submission.
Syntax: gateway [OPTIONS]
"gateway" mode options:
-e,--endpoint <SQL Gateway address> The address of the remote SQL Gateway
to connect.
-f,--file <script file> Script file that should be executed.
In this mode, the client will not
open an interactive terminal.
-h,--help Show the help message with
descriptions of all options.
-hist,--history <History file path> The file which you want to save the
command history into. If not
specified, we will auto-generate one
under your user's home directory.
-i,--init <initialization file> Script file that used to init the
session context. If get error in
execution, the sql client will exit.
Notice it's not allowed to add query
or insert into the init file.
-s,--session <session identifier> The identifier for a session.
'default' is the default identifier.
-u,--update <SQL update statement> Deprecated Experimental (for testing
only!) feature: Instructs the SQL
Client to immediately execute the
given update statement after starting
up. The process is shut down after
the statement has been submitted to
the cluster and returns an
appropriate return code. Currently,
this feature is only supported for
INSERT INTO statements that declare
the target sink table.Please use
option -f to submit update statement.
```
##########
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/SqlClient.java:
##########
@@ -56,38 +57,52 @@
* <p>- In embedded mode, the SQL CLI is tightly coupled with the executor in a common process. This
* allows for submitting jobs without having to start an additional component.
*
- * <p>- In future versions: In gateway mode, the SQL CLI client connects to the REST API of the
- * gateway and allows for managing queries via console.
- *
- * <p>For debugging in an IDE you can execute the main method of this class using: "--defaults
- * /path/to/sql-client-defaults.yaml --jar /path/to/target/flink-sql-client-*.jar"
- *
- * <p>Make sure that the FLINK_CONF_DIR environment variable is set.
+ * <p>- In gateway mode, the SQL CLI client connects to the REST API of the gateway and allows for
+ * managing queries via console.
*/
public class SqlClient {
private static final Logger LOG = LoggerFactory.getLogger(SqlClient.class);
- private final boolean isEmbedded;
+ private final boolean isGatewayMode;
private final CliOptions options;
private final Supplier<Terminal> terminalFactory;
public static final String MODE_EMBEDDED = "embedded";
public static final String MODE_GATEWAY = "gateway";
+ public static final String MODE_NONE = "";
- public SqlClient(boolean isEmbedded, CliOptions options, Supplier<Terminal> terminalFactory) {
- this.isEmbedded = isEmbedded;
+ public SqlClient(
+ boolean isGatewayMode, CliOptions options, Supplier<Terminal> terminalFactory) {
+ this.isGatewayMode = isGatewayMode;
this.options = options;
this.terminalFactory = terminalFactory;
}
private void start() {
- if (isEmbedded) {
- DefaultContext defaultContext = LocalContextUtils.buildDefaultContext(options);
+ if (isGatewayMode) {
+ CliOptions.GatewayCliOptions gatewayCliOptions = (CliOptions.GatewayCliOptions) options;
+ try (ExecutorImpl executor =
+ new ExecutorImpl(
+ DefaultContextUtils.buildDefaultContext(gatewayCliOptions),
+ gatewayCliOptions
+ .getGatewayAddress()
+ .orElseThrow(
+ () ->
+ new SqlClientException(
+ "Please specify the address of the SQL Gateway with command line option"
+ + " '-e,--endpoint <SQL Gateway address>' in the gateway mode.")))) {
Review Comment:
If the exception is thrown here, why `endpoint` might be null at `CliOptionsParser#L282`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org