You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Paul Lam <pa...@gmail.com> on 2023/05/29 11:24:17 UTC
[DISCUSS] FLIP-316: Introduce SQL Driver
Hi team,
I’d like to start a discussion about FLIP-316 [1], which introduces a SQL driver as the
default main class for Flink SQL jobs.
Currently, Flink SQL could be executed out of the box either via SQL Client/Gateway
or embedded in a Flink Java/Python program.
However, each one has its drawback:
- SQL Client/Gateway doesn’t support the application deployment mode [2]
- Flink Java/Python program requires extra work to write a non-SQL program
Therefore, I propose adding a SQL driver to act as the default main class for SQL jobs.
Please see the FLIP docs for details and feel free to comment. Thanks!
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
[2] https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541>
Best,
Paul Lam
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi ShengKai,
Good point with the ANALYZE TABLE and CALL PROCEDURE statements.
> Can we remove the jars if the job is running or gateway exits?
Yes, I think it would be okay to remove the resources after the job is submitted.
It should be Gateway’s responsibility to remove them.
> Can we use the returned rest client by ApplicationDeployer to query the job
> id? I am concerned that users don't know which job is related to the
> submitted SQL.
That should be doable, as normally we only allow one job in an application
cluster ATM.
But a more significant problem I see is that select statements are not available.
Perhaps we need to make CollectSinkFunction accept an external sink address
from SQL Gateway to get the result back from SQL Driver. WDYT?
> It seems we need to introduce a new module. Will the new module is
> available in the distribution package? I agree with Jark that we don't need
> to introduce this for table-API users and these users have their main
> class. If we want to make users write the k8s operator more easily, I think
> we should modify the k8s operator repo. If we don't need to support SQL
> files, can we make this jar only visible in the sql-gateway like we do in
> the planner loader?[1]
I rethink the relationship between SQL Driver and SQL Client with embedded
Gateway. With the help of SQL Driver, we should be able to run SQL files
with non-interactive SQL Client on K8s, just as @Biao did.
If it’s the case, I’m good with introducing a new module and making SQL Driver
an internal class and accepts JSON plans only.
WRT visibility, I lean toward making it more publicly visible and easy to integrate
with external systems. I think putting the jar in the opt folder is good. May you
elaborate a bit more about the benefit we get from an extra loader?
Best,
Paul Lam
> 2023年6月7日 17:25，Shengkai Fang <fs...@gmail.com> 写道：
>
> Hi. Paul. Thanks for your update and the update makes me understand the
> design much better.
>
> But I still have some questions about the FLIP.
>
>> For SQL Gateway, only DMLs need to be delegated to the SQL server
>> Driver. I would think about the details and update the FLIP. Do you have
> some
>> ideas already?
>
> If the applicaiton mode can not support library mode, I think we should
> only execute INSERT INTO and UPDATE/ DELETE statement in the application
> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> statements. The ANALYZE TABLE syntax need to register the statistic to the
> catalog after job finishes and the CALL PROCEDURE statement doesn't
> generate the ExecNodeGraph.
>
> * Introduce storage via option `sql-gateway.application.storage-dir`
>
> If we can not support to submit the jars through web submission, +1 to
> introduce the options to upload the files. While I think the uploader
> should be responsible to remove the uploaded jars. Can we remove the jars
> if the job is running or gateway exits?
>
> * JobID is not avaliable
>
> Can we use the returned rest client by ApplicationDeployer to query the job
> id? I am concerned that users don't know which job is related to the
> submitted SQL.
>
> * Do we need to introduce a new module named flink-table-sql-runner?
>
> It seems we need to introduce a new module. Will the new module is
> available in the distribution package? I agree with Jark that we don't need
> to introduce this for table-API users and these users have their main
> class. If we want to make users write the k8s operator more easily, I think
> we should modify the k8s operator repo. If we don't need to support SQL
> files, can we make this jar only visible in the sql-gateway like we do in
> the planner loader?[1]
>
> [1]
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
>
> Best,
> Shengkai
>
>
>
>
>
>
>
>
> Weihua Hu <hu...@gmail.com> 于2023年6月7日周三 10:52写道：
>
>> Hi,
>>
>> Thanks for updating the FLIP.
>>
>> I have two cents on the distribution of SQLs and resources.
>> 1. Should we support a common file distribution mechanism for k8s
>> application mode?
>> I have seen some issues and requirements on the mailing list.
>> In our production environment, we implement the download command in the
>> CliFrontend.
>> And automatically add an init container to the POD for file downloading.
>> The advantage of this
>> is that we can use all Flink-supported file systems to store files.
>>
>> This need more discussion. I would appreciate hearing more opinions.
>>
>> 2. In this FLIP, we distribute files in two different ways in YARN and
>> Kubernetes. Can we combine it in one way?
>> If we don't want to implement a common file distribution for k8s
>> application mode. Could we use the SQLDriver
>> to download the files both in YARN and K8S? IMO, this can reduce the cost
>> of code maintenance.
>>
>> Best,
>> Weihua
>>
>>
>> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
>>
>>> Hi Mason,
>>>
>>> Thanks for your input!
>>>
>>>> +1 for init containers or a more generalized way of obtaining arbitrary
>>>> files. File fetching isn't specific to just SQL--it also matters for
>> Java
>>>> applications if the user doesn't want to rebuild a Flink image and just
>>>> wants to modify the user application fat jar.
>>>
>>> I agree that utilizing SQL Drivers in Java applications is equally
>>> important
>>> as employing them in SQL Gateway. WRT init containers, I think most
>>> users use them just as a workaround. For example, wget a jar from the
>>> maven repo.
>>>
>>> We could implement the functionality in SQL Driver in a more graceful
>>> way and the flink-supported filesystem approach seems to be a
>>> good choice.
>>>
>>>> Also, what do you think about prefixing the config options with
>>>> `sql-driver` instead of just `sql` to be more specific?
>>>
>>> LGTM, since SQL Driver is a public interface and the options are
>>> specific to it.
>>>
>>> Best,
>>> Paul Lam
>>>
>>>> 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
>>>>
>>>> Hi Paul,
>>>>
>>>> +1 for this feature and supporting SQL file + JSON plans. We get a lot
>> of
>>>> requests to just be able to submit a SQL file, but the JSON plan
>>>> optimizations make sense.
>>>>
>>>> +1 for init containers or a more generalized way of obtaining arbitrary
>>>> files. File fetching isn't specific to just SQL--it also matters for
>> Java
>>>> applications if the user doesn't want to rebuild a Flink image and just
>>>> wants to modify the user application fat jar.
>>>>
>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>> which
>>>>> should
>>>>
>>>> be required to run Flink in production, so I guess that would be
>>> acceptable
>>>>> for most
>>>>
>>>> users. WDYT?
>>>>
>>>>
>>>> If you do go this route, it would be nice to support writing these
>> files
>>> to
>>>> S3/HDFS via Flink. This makes access control and policy management
>>> simpler.
>>>>
>>>> Also, what do you think about prefixing the config options with
>>>> `sql-driver` instead of just `sql` to be more specific?
>>>>
>>>> Best,
>>>> Mason
>>>>
>>>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
>> <mailto:
>>> paullin3280@gmail.com>> wrote:
>>>>
>>>>> Hi Jark,
>>>>>
>>>>> Thanks for your input! Please see my comments inline.
>>>>>
>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>>> DataStream API also doesn't provide a default main class for users,
>>>>>> why do we need to provide such one for SQL?
>>>>>
>>>>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>>> submitted
>>>>> via Flink CLI which actually could be DataStream/Table jobs.
>>>>>
>>>>> I think a default main class would be user-friendly which eliminates
>> the
>>>>> need
>>>>> for users to write a main class as SQLRunner in Flink K8s operator
>> [1].
>>>>>
>>>>>> I thought the proposed SqlDriver was a dedicated main class accepting
>>>>> SQL files, is
>>>>>> that correct?
>>>>>
>>>>> Both JSON plans and SQL files are accepted. SQL Gateway should use
>> JSON
>>>>> plans,
>>>>> while CLI users may use either JSON plans or SQL files.
>>>>>
>>>>> Please see the updated FLIP[2] for more details.
>>>>>
>>>>>> Personally, I prefer the way of init containers which doesn't depend
>> on
>>>>>> additional components.
>>>>>> This can reduce the moving parts of a production environment.
>>>>>> Depending on a distributed file system makes the testing, demo, and
>>> local
>>>>>> setup harder than init containers.
>>>>>
>>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>> which
>>>>> should
>>>>> be required to run Flink in production, so I guess that would be
>>>>> acceptable for most
>>>>> users. WDYT?
>>>>>
>>>>> WRT testing, demo, and local setups, I think we could support the
>> local
>>>>> filesystem
>>>>> scheme i.e. file://** as the state backends do. It works as long as
>> SQL
>>>>> Gateway
>>>>> and JobManager(or SQL Driver) can access the resource directory
>>> (specified
>>>>> via
>>>>> `sql-gateway.application.storage-dir`).
>>>>>
>>>>> Thanks!
>>>>>
>>>>> [1]
>>>>>
>>>
>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
>>>>> [2]
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>> [3]
>>>>>
>>>
>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>>> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
>>>>>>
>>>>>> Hi Paul,
>>>>>>
>>>>>> Thanks for your reply. I left my comments inline.
>>>>>>
>>>>>>> As the FLIP said, it’s good to have a default main class for Flink
>>> SQLs,
>>>>>>> which allows users to submit Flink SQLs in the same way as
>> DataStream
>>>>>>> jobs, or else users need to write their own main class.
>>>>>>
>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>>> DataStream API also doesn't provide a default main class for users,
>>>>>> why do we need to provide such one for SQL?
>>>>>>
>>>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>>>> SessionState? If not, we could make SQL Driver accepts two
>> serialized
>>>>>>> formats:
>>>>>>
>>>>>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
>> the
>>>>>> proposed SqlDriver was a dedicated main class accepting SQL files, is
>>>>>> that correct?
>>>>>> If true, we have to ship the SessionState for this case which is a
>>> large
>>>>>> work.
>>>>>> I think we just need a JsonPlanDriver which is a main class that
>>> accepts
>>>>>> JsonPlan as the parameter.
>>>>>>
>>>>>>
>>>>>>> The common solutions I know is to use distributed file systems or
>> use
>>>>>>> init containers to localize the resources.
>>>>>>
>>>>>> Personally, I prefer the way of init containers which doesn't depend
>> on
>>>>>> additional components.
>>>>>> This can reduce the moving parts of a production environment.
>>>>>> Depending on a distributed file system makes the testing, demo, and
>>> local
>>>>>> setup harder than init containers.
>>>>>>
>>>>>> Best,
>>>>>> Jark
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
>> <mailto:
>>> paullin3280@gmail.com> <mailto:
>>>>> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>>>>>>
>>>>>>> The FLIP is in the early phase and some details are not included,
>> but
>>>>>>> fortunately, we got lots of valuable ideas from the discussion.
>>>>>>>
>>>>>>> Thanks to everyone who joined the dissuasion!
>>>>>>> @Weihua @Shanmon @Shengkai @Biao @Jark
>>>>>>>
>>>>>>> This weekend I’m gonna revisit and update the FLIP, adding more
>>>>>>> details. Hopefully, we can further align our opinions.
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
>>> paullin3280@gmail.com>> 写道：
>>>>>>>>
>>>>>>>> Hi Jark,
>>>>>>>>
>>>>>>>> Thanks a lot for your input!
>>>>>>>>
>>>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>> still
>>>>>>>>> necessary to support SQL Driver?
>>>>>>>>
>>>>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
>>> simplify
>>>>>>>> Flink SQL execution with Flink CLI.
>>>>>>>>
>>>>>>>> As the FLIP said, it’s good to have a default main class for Flink
>>>>> SQLs,
>>>>>>>> which allows users to submit Flink SQLs in the same way as
>> DataStream
>>>>>>>> jobs, or else users need to write their own main class.
>>>>>>>>
>>>>>>>>> SQL Driver needs to serialize SessionState which is very
>> challenging
>>>>>>>>> but not detailed covered in the FLIP.
>>>>>>>>
>>>>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>>>>> SessionState? If not, we could make SQL Driver accepts two
>> serialized
>>>>>>>> formats:
>>>>>>>>
>>>>>>>> - SQL files for user-facing public usage
>>>>>>>> - ExecNodeGraph for internal usage
>>>>>>>>
>>>>>>>> It’s kind of similar to the relationship between job jars and
>>>>> jobgraphs.
>>>>>>>>
>>>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>>> true?
>>>>>>> Is it
>>>>>>>>> possible to support it?
>>>>>>>>
>>>>>>>> Yes, K8s doesn’t distribute any files. It’s the users’
>> responsibility
>>>>> to
>>>>>>> make
>>>>>>>> sure the resources are accessible in the containers. The common
>>>>> solutions
>>>>>>>> I know is to use distributed file systems or use init containers to
>>>>>>> localize the
>>>>>>>> resources.
>>>>>>>>
>>>>>>>> Now I lean toward introducing a fs to do the distribution job.
>> WDYT?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Paul Lam
>>>>>>>>
>>>>>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
>> imjark@gmail.com>
>>> <mailto:imjark@gmail.com <ma...@gmail.com>>
>>>>> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
>>> imjark@gmail.com <ma...@gmail.com>>>>
>>>>>>> 写道：
>>>>>>>>>
>>>>>>>>> Hi Paul,
>>>>>>>>>
>>>>>>>>> Thanks for starting this discussion. I like the proposal! This is
>> a
>>>>>>>>> frequently requested feature!
>>>>>>>>>
>>>>>>>>> I agree with Shengkai that ExecNodeGraph as the submission object
>>> is a
>>>>>>>>> better idea than SQL file. To be more specific, it should be
>>>>>>> JsonPlanGraph
>>>>>>>>> or CompiledPlan which is the serializable representation.
>>> CompiledPlan
>>>>>>> is a
>>>>>>>>> clear separation between compiling/optimization/validation and
>>>>>>> execution.
>>>>>>>>> This can keep the validation and metadata accessing still on the
>>>>>>> SQLGateway
>>>>>>>>> side. This allows SQLGateway to leverage some metadata caching and
>>> UDF
>>>>>>> JAR
>>>>>>>>> caching for better compiling performance.
>>>>>>>>>
>>>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>> still
>>>>>>>>> necessary to support SQL Driver? Regarding non-interactive SQL
>> jobs,
>>>>>>> users
>>>>>>>>> can use the Table API program for application mode. SQL Driver
>> needs
>>>>> to
>>>>>>>>> serialize SessionState which is very challenging but not detailed
>>>>>>> covered
>>>>>>>>> in the FLIP.
>>>>>>>>>
>>>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>>> true?
>>>>>>> Is it
>>>>>>>>> possible to support it?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Jark
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
>>> <ma...@gmail.com> <mailto:
>>>>> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:
>>> paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Weihua,
>>>>>>>>>>
>>>>>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
>>>>>>> challenging
>>>>>>>>>> parts of this FLIP.
>>>>>>>>>>
>>>>>>>>>> Web submission is not enabled in application mode currently as
>> you
>>>>>>> said,
>>>>>>>>>> but it could be changed if we have good reasons.
>>>>>>>>>>
>>>>>>>>>> What do you think about introducing a distributed storage for SQL
>>>>>>> Gateway?
>>>>>>>>>>
>>>>>>>>>> We could make use of Flink file systems [1] to distribute the SQL
>>>>>>> Gateway
>>>>>>>>>> generated resources, that should solve the problem at its root
>>> cause.
>>>>>>>>>>
>>>>>>>>>> Users could specify Flink-supported file systems to ship files.
>>> It’s
>>>>>>> only
>>>>>>>>>> required when using SQL Gateway with K8s application mode.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>> <
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>> <
>>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>> <
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>
>>>>>>> <
>>>>>>>
>>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>> <
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>> <
>>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>> <
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Paul Lam
>>>>>>>>>>
>>>>>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
>>> huweihua.ckl@gmail.com> <mailto:
>>>>> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
>>>>>>>>>>>
>>>>>>>>>>> Thanks Paul for your reply.
>>>>>>>>>>>
>>>>>>>>>>> SQLDriver looks good to me.
>>>>>>>>>>>
>>>>>>>>>>> 2. Do you mean a pass the SQL string a configuration or a
>> program
>>>>>>>>>> argument?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I brought this up because we were unable to pass the SQL file to
>>>>> Flink
>>>>>>>>>>> using Kubernetes mode.
>>>>>>>>>>> For DataStream/Python users, they need to prepare their images
>> for
>>>>> the
>>>>>>>>>> jars
>>>>>>>>>>> and dependencies.
>>>>>>>>>>> But for SQL users, they can use a common image to run different
>>> SQL
>>>>>>>>>> queries
>>>>>>>>>>> if there are no other udf requirements.
>>>>>>>>>>> It would be great if the SQL query and image were not bound.
>>>>>>>>>>>
>>>>>>>>>>> Using strings is a way to decouple these, but just as you
>>> mentioned,
>>>>>>> it's
>>>>>>>>>>> not easy to pass complex SQL.
>>>>>>>>>>>
>>>>>>>>>>>> use web submission
>>>>>>>>>>> AFAIK, we can not use web submission in the Application mode.
>>> Please
>>>>>>>>>>> correct me if I'm wrong.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Weihua
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <paullin3280@gmail.com
>>> <ma...@gmail.com>
>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Biao,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for your comments!
>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink
>>> SQL
>>>>>>> jobs
>>>>>>>>>>>> in
>>>>>>>>>>>>> Application mode? More specifically, if we use SQL
>>> client/gateway
>>>>> to
>>>>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
>>>>> flink
>>>>>>> to
>>>>>>>>>>>> use
>>>>>>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for pointing it out. I think only DMLs would be executed
>>> via
>>>>>>> SQL
>>>>>>>>>>>> Driver.
>>>>>>>>>>>> I'll add the scope to the FLIP.
>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>>> trivial as
>>>>>>>>>>>> we
>>>>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can
>> be
>>>>>>> more
>>>>>>>>>>>>> complicated as Shengkai said.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Your input is very informative. I’m thinking about using web
>>>>>>> submission,
>>>>>>>>>>>> but it requires exposing the JobManager port which could also
>> be
>>> a
>>>>>>>>>> problem
>>>>>>>>>>>> on K8s.
>>>>>>>>>>>>
>>>>>>>>>>>> Another approach is to explicitly require a distributed storage
>>> to
>>>>>>> ship
>>>>>>>>>>>> files,
>>>>>>>>>>>> but we may need a new deployment executor for that.
>>>>>>>>>>>>
>>>>>>>>>>>> What do you think of these two approaches?
>>>>>>>>>>>>
>>>>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
>>> some
>>>>>>>>>>>>> unserializable fields
>>>>>>>>>>>>> like
>>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>>>>> It
>>>>>>>>>>>>> may be worthwhile to add more details about the serialization
>>>>> part.
>>>>>>>>>>>>
>>>>>>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as
>>>>>>> Shengkai
>>>>>>>>>>>> mentioned, do we eliminate the need for serialization of
>>>>>>> SessionState?
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Paul Lam
>>>>>>>>>>>>
>>>>>>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
>>> biaogeng7@gmail.com> <mailto:
>>>>> biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Paul for the proposal!I believe it would be very useful
>>> for
>>>>>>>>>> flink
>>>>>>>>>>>>> users.
>>>>>>>>>>>>> After reading the FLIP, I have some questions:
>>>>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink
>>> SQL
>>>>>>> jobs
>>>>>>>>>>>> in
>>>>>>>>>>>>> Application mode? More specifically, if we use SQL
>>> client/gateway
>>>>> to
>>>>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
>>>>> flink
>>>>>>> to
>>>>>>>>>>>> use
>>>>>>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>>> trivial as
>>>>>>>>>>>> we
>>>>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can
>> be
>>>>>>> more
>>>>>>>>>>>>> complicated as Shengkai said. I have implemented a simple POC
>>>>>>>>>>>>> <
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>> <
>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>
>>>>> <
>>>>>
>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>> <
>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>
>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> based on SQL client before(i.e. consider the SQL client which
>>>>>>> supports
>>>>>>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One
>>> problem
>>>>> I
>>>>>>>>>> have
>>>>>>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s
>> side.
>>>>>>>>>> Without
>>>>>>>>>>>>> such support, users have to modify the initContainer or
>> rebuild
>>> a
>>>>>>> new
>>>>>>>>>> K8s
>>>>>>>>>>>>> image every time to fetch the SQL file. Like the flink k8s
>>>>> operator,
>>>>>>>>>> one
>>>>>>>>>>>>> workaround is to utilize the flink config(transforming the SQL
>>>>> file
>>>>>>> to
>>>>>>>>>> a
>>>>>>>>>>>>> escaped string like Weihua mentioned) which will be converted
>>> to a
>>>>>>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than
>>> 1MB
>>>>>>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/
>> <
>>> https://kubernetes.io/docs/concepts/configuration/configmap/> <
>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ <
>>> https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
>>>>>>> Not
>>>>>>>>>>>> sure
>>>>>>>>>>>>> if we have better solutions.
>>>>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
>>> some
>>>>>>>>>>>>> unserializable fields
>>>>>>>>>>>>> like
>>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>>>>> It
>>>>>>>>>>>>> may be worthwhile to add more details about the serialization
>>>>> part.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Biao Geng
>>>>>>>>>>>>>
>>>>>>>>>>>>> Paul Lam <paullin3280@gmail.com <mailto:paullin3280@gmail.com
>>>
>>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>>> 于2023年5月31日周三 11:49写道：
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Weihua,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
>> Job.
>>>>> (Not
>>>>>>>>>>>>>> strong,
>>>>>>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
>>>>>>> following
>>>>>>>>>>>>>> reasons FYI:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>>>>>>>>>>> 2. Flink program's main class is sort of like Driver in JDBC
>>>>> which
>>>>>>>>>>>>>> translates SQLs into
>>>>>>>>>>>>>> databases specific languages.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we
>>> need
>>>>>>> to
>>>>>>>>>>>>>> prepare
>>>>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode,
>> which
>>>>> may
>>>>>>> be
>>>>>>>>>> a
>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>> cumbersome.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you mean a pass the SQL string a configuration or a
>> program
>>>>>>>>>> argument?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I thought it might be convenient for testing propose, but not
>>>>>>>>>>>> recommended
>>>>>>>>>>>>>> for production,
>>>>>>>>>>>>>> cause Flink SQLs could be complicated and involves lots of
>>>>>>> characters
>>>>>>>>>>>> that
>>>>>>>>>>>>>> need to escape.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>>>>>>> "run-application"
>>>>>>>>>>>>>>> command. Does that mean we need to perform automatic
>> detection
>>>>> in
>>>>>>>>>>>> Flink?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes! It’s like running a PyFlink job with the following
>>> command:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>> ./bin/flink run \
>>>>>>>>>>>>>> --pyModule table.word_count \
>>>>>>>>>>>>>> --pyFiles examples/python/table
>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
>>> Driver
>>>>>>>>>>>>>> automatically.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>> <
>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>
>>>>> <
>>>>>
>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>> <
>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>
>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Paul Lam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <mailto:
>>> huweihua.ckl@gmail.com> <mailto:
>>>>> huweihua.ckl@gmail.com>> 写道：
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks Paul for the proposal.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have a few questions.
>>>>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
>> Job.
>>>>> (Not
>>>>>>>>>>>>>> strong,
>>>>>>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we
>>> need
>>>>>>> to
>>>>>>>>>>>>>> prepare
>>>>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode,
>> which
>>>>> may
>>>>>>> be
>>>>>>>>>> a
>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>> cumbersome.
>>>>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>>>>>>> "run-application"
>>>>>>>>>>>>>>> command. Does that mean we need to perform automatic
>> detection
>>>>> in
>>>>>>>>>>>> Flink?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Weihua
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>>> paullin3280@gmail.com
>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi team,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
>>>>>>> introduces
>>>>>>>>>> a
>>>>>>>>>>>>>> SQL
>>>>>>>>>>>>>>>> driver as the
>>>>>>>>>>>>>>>> default main class for Flink SQL jobs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Currently, Flink SQL could be executed out of the box
>> either
>>>>> via
>>>>>>> SQL
>>>>>>>>>>>>>>>> Client/Gateway
>>>>>>>>>>>>>>>> or embedded in a Flink Java/Python program.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, each one has its drawback:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application
>>> deployment
>>>>>>> mode
>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>> - Flink Java/Python program requires extra work to write a
>>>>>>> non-SQL
>>>>>>>>>>>>>> program
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the
>>> default
>>>>>>> main
>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>> for SQL jobs.
>>>>>>>>>>>>>>>> Please see the FLIP docs for details and feel free to
>>> comment.
>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Paul Lam
>>>
>>>
>>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Venkatakrishnan Sowrirajan <vs...@asu.edu>.
Hi Paul Lam,
Is this FLIP still active? I am not able to find the vote for this FLIP and
any corresponding JIRA or Github PR with POC. Can you please share more on
the current status of this FLIP? Thanks!
Regards
Venkata krishnan
On Fri, Jan 5, 2024 at 2:40 AM Márton Balassi <ba...@gmail.com>
wrote:
> Thanks, Paul.
>
> Ferenc and I have been looking into unblocking the Kubernetes path via an
> updated implementation for FLINK-28915 to ship the jars conveniently there.
> You can expect an updated PR there next week. Looking forward to your
> findings in the YARN POC.
>
> On Mon, Dec 11, 2023 at 4:01 AM Paul Lam <pa...@gmail.com> wrote:
>
> > Hi Ferenc,
> >
> > Sorry for my late reply.
> >
> > > Is any active work happening on this FLIP? As far as I see there
> > > are blockers that needs to happen first to implement regarding
> > > artifact distribution.
> >
> > You’re right. There’s a block in K8s application mode, but none in
> > YARN application. I’m doing a POC on YARN application mode
> > before starting a vote thread.
> >
> > I’ve been busy lately, but the FLIP is active for sure. The situation
> > will change in a couple of weeks.
> >
> > Thank you for reaching out! I’ll let you know when the POC is completed.
> >
> > Best,
> > Paul Lam
> >
> > > 2023年11月21日 06:01，Ferenc Csaky <fe...@pm.me.INVALID> 写道：
> > >
> > > Hello devs,
> > >
> > > Is any active work happening on this FLIP? As far as I see there
> > > are blockers that needs to happen first to implement regarding
> > > artifact distribution.
> > >
> > > Is this work in halt completetly or some efforts are going into
> > > resolve the blockers first or something?
> > >
> > > Our platform would benefit this feature a lot, we have a kind of
> > > working custom implementation at the moment, but it is uniquely
> > > adapted to our app and platform.
> > >
> > > I could help out to move this forward.
> > >
> > > Best,
> > > Ferenc
> > >
> > >
> > >
> > > On Friday, June 30th, 2023 at 04:53, Paul Lam <paullin3280@gmail.com
> > <ma...@gmail.com>> wrote:
> > >
> > >
> > >>
> > >>
> > >> Hi Jing,
> > >>
> > >> Thanks for your input!
> > >>
> > >>> Would you like to add
> > >>> one section to describe(better with script/code example) how to use
> it
> > in
> > >>> these two scenarios from users' perspective?
> > >>
> > >>
> > >> OK. I’ll update the FLIP with the code snippet after I get the POC
> > branch done.
> > >>
> > >>> NIT: the pictures have transparent background when readers click on
> > it. It
> > >>> would be great if you can replace them with pictures with white
> > background.
> > >>
> > >>
> > >> Fixed. Thanks for pointing that out :)
> > >>
> > >> Best,
> > >> Paul Lam
> > >>
> > >>> 2023年6月27日 06:51，Jing Ge jing@ververica.com.INVALID 写道：
> > >>>
> > >>> Hi Paul,
> > >>>
> > >>> Thanks for driving it and thank you all for the informative
> > discussion! The
> > >>> FLIP is in good shape now. As described in the FLIP, SQL Driver will
> be
> > >>> mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway
> > in
> > >>> application mode and 2. external system integration. Would you like
> to
> > add
> > >>> one section to describe(better with script/code example) how to use
> it
> > in
> > >>> these two scenarios from users' perspective?
> > >>>
> > >>> NIT: the pictures have transparent background when readers click on
> > it. It
> > >>> would be great if you can replace them with pictures with white
> > background.
> > >>>
> > >>> Best regards,
> > >>> Jing
> > >>>
> > >>> On Mon, Jun 26, 2023 at 1:31 PM Paul Lam <paullin3280@gmail.com
> > <ma...@gmail.com> mailto:paullin3280@gmail.com> wrote:
> > >>>
> > >>>> Hi Shengkai,
> > >>>>
> > >>>>> * How can we ship the json plan to the JobManager?
> > >>>>
> > >>>> The Flink K8s module should be responsible for file distribution. We
> > could
> > >>>> introduce
> > >>>> an option like `kubernetes.storage.dir`. For each flink cluster,
> there
> > >>>> would be a
> > >>>> dedicated subdirectory, with the pattern like
> > >>>> `${kubernetes.storage.dir}/${cluster-id}`.
> > >>>>
> > >>>> All resources-related options (e.g. pipeline jars, json plans) that
> > are
> > >>>> configured with
> > >>>> scheme `file://` <file://`/> <file://`/> <file:///%60 <file:///%60>>
> > would be uploaded to the resource directory
> > >>>> and downloaded to the
> > >>>> jobmanager, before SQL Driver accesses the files with the original
> > >>>> filenames.
> > >>>>
> > >>>>> * Classloading strategy
> > >>>>
> > >>>> We could directly specify the SQL Gateway jar as the jar file in
> > >>>> PackagedProgram.
> > >>>> It would be treated like a normal user jar and the SQL Driver is
> > loaded
> > >>>> into the user
> > >>>> classloader. WDYT?
> > >>>>
> > >>>>> * Option `$internal.sql-gateway.driver.sql-config` is string type
> > >>>>> I think it's better to use Map type here
> > >>>>
> > >>>> By Map type configuration, do you mean a nested map that contains
> all
> > >>>> configurations?
> > >>>>
> > >>>> I hope I've explained myself well, it’s a file that contains the
> > extra SQL
> > >>>> configurations, which would be shipped to the jobmanager.
> > >>>>
> > >>>>> * PoC branch
> > >>>>
> > >>>> Sure. I’ll let you know once I get the job done.
> > >>>>
> > >>>> Best,
> > >>>> Paul Lam
> > >>>>
> > >>>>> 2023年6月26日 14:27，Shengkai Fang <fskmine@gmail.com <mailto:
> > fskmine@gmail.com> mailto:fskmine@gmail.com> 写道：
> > >>>>>
> > >>>>> Hi, Paul.
> > >>>>>
> > >>>>> Thanks for your update. I have a few questions about the new
> design:
> > >>>>>
> > >>>>> * How can we ship the json plan to the JobManager?
> > >>>>>
> > >>>>> The current design only exposes an option about the URL of the json
> > >>>>> plan. It seems the gateway is responsible to upload to an external
> > stroage.
> > >>>>> Can we reuse the PipelineOptions.JARS to ship to the remote
> > filesystem?
> > >>>>>
> > >>>>> * Classloading strategy
> > >>>>>
> > >>>>> Currently, the Driver is in the sql-gateway package. It means the
> > Driver
> > >>>>> is not in the JM's classpath directly. Because the sql-gateway jar
> > is now
> > >>>>> in the opt directory rather than lib directory. It may need to add
> > the
> > >>>>> external dependencies as Python does[1]. BTW, I think it's better
> to
> > move
> > >>>>> the Driver into the flink-table-runtime package, which is much
> > easier to
> > >>>>> find(Sorry for the wrong opinion before).
> > >>>>>
> > >>>>> * Option `$internal.sql-gateway.driver.sql-config` is string type
> > >>>>>
> > >>>>> I think it's better to use Map type here
> > >>>>>
> > >>>>> * PoC branch
> > >>>>>
> > >>>>> Because this FLIP involves many modules, do you have a PoC branch
> to
> > >>>>> verify it does work?
> > >>>>>
> > >>>>> Best,
> > >>>>> Shengkai
> > >>>>>
> > >>>>> [1]
> > >>>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java*L940https:/*github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java*L940__;Iy8j!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHSM_l0zt$
> > >>>>> <
> > >>>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java*L940https:/*github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java*L940__;Iy8j!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHSM_l0zt$
> > >>>>>
> > >>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> > mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>
> > >>>>> 于2023年6月19日周一 14:09写道：
> > >>>>> Hi Shengkai,
> > >>>>>
> > >>>>> Sorry for my late reply. It took me some time to update the FLIP.
> > >>>>>
> > >>>>> In the latest FLIP design, SQL Driver is placed in
> flink-sql-gateway
> > >>>>> module. PTAL.
> > >>>>>
> > >>>>> The FLIP does not cover details about the K8s file distribution,
> but
> > its
> > >>>>> general usage would
> > >>>>> be very much the same as YARN setups. We could make follow-up
> > >>>>> discussions in the jira
> > >>>>> tickets.
> > >>>>>
> > >>>>> Best,
> > >>>>> Paul Lam
> > >>>>>
> > >>>>>> 2023年6月12日 15:29，Shengkai Fang <fskmine@gmail.com <mailto:
> > fskmine@gmail.com> mailto:fskmine@gmail.com <mailto:
> > >>>>>> fskmine@gmail.com <ma...@gmail.com> mailto:
> > fskmine@gmail.com>> 写道：
> > >>>>>>
> > >>>>>>> If it’s the case, I’m good with introducing a new module and
> making
> > >>>>>>> SQL Driver
> > >>>>>>> an internal class and accepts JSON plans only.
> > >>>>>>
> > >>>>>> I rethink this again and again. I think it's better to move the
> > >>>>>> SqlDriver into the sql-gateway module because the sql client
> relies
> > on the
> > >>>>>> sql-gateway to submit the sql and the sql-gateway has the ability
> to
> > >>>>>> generate the ExecNodeGraph now. +1 to support accepting JSON plans
> > only.
> > >>>>>>
> > >>>>>> * Upload configuration through command line parameter
> > >>>>>>
> > >>>>>> ExecNodeGraph only contains the job's information but it doesn't
> > >>>>>> contain the checkpoint dir, checkpoint interval, execution mode
> and
> > so on.
> > >>>>>> So I think we should also upload the configuration.
> > >>>>>>
> > >>>>>> * KubernetesClusterDescripter and
> > >>>>>> KubernetesApplicationClusterEntrypoint are responsible for the jar
> > >>>>>> upload/download
> > >>>>>>
> > >>>>>> +1 for the change.
> > >>>>>>
> > >>>>>> Could you update the FLIP about the current discussion?
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Shengkai
> > >>>>>>
> > >>>>>> Yang Wang <wangyang0918@apache.org <mailto:
> wangyang0918@apache.org>
> > mailto:wangyang0918@apache.org <mailto:wangyang0918@apache.orgmailto:
> > wangyang0918@apache.org>>
> > >>>>>> 于2023年6月12日周一 11:41写道：
> > >>>>>> Sorry for the late reply. I am in favor of introducing such a
> > built-in
> > >>>>>> resource localization mechanism
> > >>>>>> based on Flink FileSystem. Then FLINK-28915[1] could be the second
> > step
> > >>>>>> which will download
> > >>>>>> the jars and dependencies to the JobManager/TaskManager local
> > directory
> > >>>>>> before working.
> > >>>>>>
> > >>>>>> The first step could be done in another ticket in Flink. Or some
> > >>>>>> external
> > >>>>>> Flink jobs management system
> > >>>>>> could also take care of this.
> > >>>>>>
> > >>>>>> [1].
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-28915__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHYzI8Wuk$
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-28915__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHYzI8Wuk$
> <
> > >>>>>>
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-28915__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHYzI8Wuk$
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-28915__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHYzI8Wuk$
> >
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Yang
> > >>>>>>
> > >>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> > mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>
> > >>>>>> 于2023年6月9日周五 17:39写道：
> > >>>>>>
> > >>>>>>> Hi Mason,
> > >>>>>>>
> > >>>>>>> I get your point. I'm increasingly feeling the need to introduce
> a
> > >>>>>>> built-in
> > >>>>>>> file distribution mechanism for flink-kubernetes module, just
> like
> > >>>>>>> Spark
> > >>>>>>> does with `spark.kubernetes.file.upload.path` [1].
> > >>>>>>>
> > >>>>>>> I’m assuming the workflow is as follows:
> > >>>>>>>
> > >>>>>>> - KubernetesClusterDescripter uploads all local resources to a
> > remote
> > >>>>>>> storage via Flink filesystem (skips if the resources are already
> > >>>>>>> remote).
> > >>>>>>> - KubernetesApplicationClusterEntrypoint downloads the resources
> > >>>>>>> and put them in the classpath during startup.
> > >>>>>>>
> > >>>>>>> I wouldn't mind splitting it into another FLIP to ensure that
> > >>>>>>> everything is
> > >>>>>>> done correctly.
> > >>>>>>>
> > >>>>>>> cc'ed @Yang to gather more opinions.
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://spark.apache.org/docs/latest/running-on-kubernetes.html*dependency-management__;Iw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHVSDNF8A$
> >
> https://urldefense.com/v3/__https://spark.apache.org/docs/latest/running-on-kubernetes.html*dependency-management__;Iw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHVSDNF8A$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://spark.apache.org/docs/latest/running-on-kubernetes.html*dependency-management__;Iw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHVSDNF8A$
> >
> https://urldefense.com/v3/__https://spark.apache.org/docs/latest/running-on-kubernetes.html*dependency-management__;Iw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHVSDNF8A$
> > >>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>> 2023年6月8日 12:15，Mason Chen <mas.chen6345@gmail.com <mailto:
> > mas.chen6345@gmail.com> mailto:mas.chen6345@gmail.com <mailto:
> > >>>>>>> mas.chen6345@gmail.com <ma...@gmail.com> mailto:
> > mas.chen6345@gmail.com>> 写道：
> > >>>>>>>
> > >>>>>>> Hi Paul,
> > >>>>>>>
> > >>>>>>> Thanks for your response!
> > >>>>>>>
> > >>>>>>> I agree that utilizing SQL Drivers in Java applications is
> equally
> > >>>>>>> important
> > >>>>>>>
> > >>>>>>> as employing them in SQL Gateway. WRT init containers, I think
> most
> > >>>>>>> users use them just as a workaround. For example, wget a jar from
> > the
> > >>>>>>> maven repo.
> > >>>>>>>
> > >>>>>>> We could implement the functionality in SQL Driver in a more
> > graceful
> > >>>>>>> way and the flink-supported filesystem approach seems to be a
> > >>>>>>> good choice.
> > >>>>>>>
> > >>>>>>> My main point is: can we solve the problem with a design agnostic
> > of
> > >>>>>>> SQL
> > >>>>>>> and Stream API? I mentioned a use case where this ability is
> useful
> > >>>>>>> for
> > >>>>>>> Java or Stream API applications. Maybe this is even a non-goal to
> > >>>>>>> your FLIP
> > >>>>>>> since you are focusing on the driver entrypoint.
> > >>>>>>>
> > >>>>>>> Jark mentioned some optimizations:
> > >>>>>>>
> > >>>>>>> This allows SQLGateway to leverage some metadata caching and UDF
> > JAR
> > >>>>>>> caching for better compiling performance.
> > >>>>>>>
> > >>>>>>> It would be great to see this even outside the SQLGateway (i.e.
> UDF
> > >>>>>>> JAR
> > >>>>>>> caching).
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Mason
> > >>>>>>>
> > >>>>>>> On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskmine@gmail.com
> > <ma...@gmail.com> mailto:fskmine@gmail.com
> > >>>>>>> <mailto:fskmine@gmail.com mailto:fskmine@gmail.com>> wrote:
> > >>>>>>>
> > >>>>>>> Hi. Paul. Thanks for your update and the update makes me
> understand
> > >>>>>>> the
> > >>>>>>> design much better.
> > >>>>>>>
> > >>>>>>> But I still have some questions about the FLIP.
> > >>>>>>>
> > >>>>>>> For SQL Gateway, only DMLs need to be delegated to the SQL server
> > >>>>>>> Driver. I would think about the details and update the FLIP. Do
> you
> > >>>>>>> have
> > >>>>>>>
> > >>>>>>> some
> > >>>>>>>
> > >>>>>>> ideas already?
> > >>>>>>>
> > >>>>>>> If the applicaiton mode can not support library mode, I think we
> > >>>>>>> should
> > >>>>>>> only execute INSERT INTO and UPDATE/ DELETE statement in the
> > >>>>>>> application
> > >>>>>>> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> > >>>>>>> statements. The ANALYZE TABLE syntax need to register the
> statistic
> > >>>>>>> to the
> > >>>>>>> catalog after job finishes and the CALL PROCEDURE statement
> doesn't
> > >>>>>>> generate the ExecNodeGraph.
> > >>>>>>>
> > >>>>>>> * Introduce storage via option
> > `sql-gateway.application.storage-dir`
> > >>>>>>>
> > >>>>>>> If we can not support to submit the jars through web submission,
> > +1 to
> > >>>>>>> introduce the options to upload the files. While I think the
> > uploader
> > >>>>>>> should be responsible to remove the uploaded jars. Can we remove
> > the
> > >>>>>>> jars
> > >>>>>>> if the job is running or gateway exits?
> > >>>>>>>
> > >>>>>>> * JobID is not avaliable
> > >>>>>>>
> > >>>>>>> Can we use the returned rest client by ApplicationDeployer to
> query
> > >>>>>>> the job
> > >>>>>>> id? I am concerned that users don't know which job is related to
> > the
> > >>>>>>> submitted SQL.
> > >>>>>>>
> > >>>>>>> * Do we need to introduce a new module named
> > flink-table-sql-runner?
> > >>>>>>>
> > >>>>>>> It seems we need to introduce a new module. Will the new module
> is
> > >>>>>>> available in the distribution package? I agree with Jark that we
> > >>>>>>> don't need
> > >>>>>>> to introduce this for table-API users and these users have their
> > main
> > >>>>>>> class. If we want to make users write the k8s operator more
> > easily, I
> > >>>>>>> think
> > >>>>>>> we should modify the k8s operator repo. If we don't need to
> support
> > >>>>>>> SQL
> > >>>>>>> files, can we make this jar only visible in the sql-gateway like
> we
> > >>>>>>> do in
> > >>>>>>> the planner loader?[1]
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java*L95__;Iw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRwUE8FA$
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java*L95__;Iw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRwUE8FA$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java*L95__;Iw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRwUE8FA$
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java*L95__;Iw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRwUE8FA$
> > >>>>
> > >>>>>>> Best,
> > >>>>>>> Shengkai
> > >>>>>>>
> > >>>>>>> Weihua Hu <huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com
> >
> > mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> > huweihua.ckl@gmail.com>>
> > >>>>>>> 于2023年6月7日周三 10:52写道：
> > >>>>>>>
> > >>>>>>> Hi,
> > >>>>>>>
> > >>>>>>> Thanks for updating the FLIP.
> > >>>>>>>
> > >>>>>>> I have two cents on the distribution of SQLs and resources.
> > >>>>>>> 1. Should we support a common file distribution mechanism for k8s
> > >>>>>>> application mode?
> > >>>>>>> I have seen some issues and requirements on the mailing list.
> > >>>>>>> In our production environment, we implement the download command
> in
> > >>>>>>> the
> > >>>>>>> CliFrontend.
> > >>>>>>> And automatically add an init container to the POD for file
> > >>>>>>>
> > >>>>>>> downloading.
> > >>>>>>>
> > >>>>>>> The advantage of this
> > >>>>>>> is that we can use all Flink-supported file systems to store
> files.
> > >>>>>>>
> > >>>>>>> This need more discussion. I would appreciate hearing more
> > opinions.
> > >>>>>>>
> > >>>>>>> 2. In this FLIP, we distribute files in two different ways in
> YARN
> > and
> > >>>>>>> Kubernetes. Can we combine it in one way?
> > >>>>>>> If we don't want to implement a common file distribution for k8s
> > >>>>>>> application mode. Could we use the SQLDriver
> > >>>>>>> to download the files both in YARN and K8S? IMO, this can reduce
> > the
> > >>>>>>>
> > >>>>>>> cost
> > >>>>>>>
> > >>>>>>> of code maintenance.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Weihua
> > >>>>>>>
> > >>>>>>> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com
> > <ma...@gmail.com> mailto:paullin3280@gmail.com
> > >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> > wrote:
> > >>>>>>>
> > >>>>>>> Hi Mason,
> > >>>>>>>
> > >>>>>>> Thanks for your input!
> > >>>>>>>
> > >>>>>>> +1 for init containers or a more generalized way of obtaining
> > >>>>>>>
> > >>>>>>> arbitrary
> > >>>>>>>
> > >>>>>>> files. File fetching isn't specific to just SQL--it also matters
> > for
> > >>>>>>>
> > >>>>>>> Java
> > >>>>>>>
> > >>>>>>> applications if the user doesn't want to rebuild a Flink image
> and
> > >>>>>>>
> > >>>>>>> just
> > >>>>>>>
> > >>>>>>> wants to modify the user application fat jar.
> > >>>>>>>
> > >>>>>>> I agree that utilizing SQL Drivers in Java applications is
> equally
> > >>>>>>> important
> > >>>>>>> as employing them in SQL Gateway. WRT init containers, I think
> most
> > >>>>>>> users use them just as a workaround. For example, wget a jar from
> > the
> > >>>>>>> maven repo.
> > >>>>>>>
> > >>>>>>> We could implement the functionality in SQL Driver in a more
> > graceful
> > >>>>>>> way and the flink-supported filesystem approach seems to be a
> > >>>>>>> good choice.
> > >>>>>>>
> > >>>>>>> Also, what do you think about prefixing the config options with
> > >>>>>>> `sql-driver` instead of just `sql` to be more specific?
> > >>>>>>>
> > >>>>>>> LGTM, since SQL Driver is a public interface and the options are
> > >>>>>>> specific to it.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>> 2023年6月6日 06:30，Mason Chen <mas.chen6345@gmail.com <mailto:
> > mas.chen6345@gmail.com> mailto:mas.chen6345@gmail.com <mailto:
> > >>>>>>> mas.chen6345@gmail.com <ma...@gmail.com> mailto:
> > mas.chen6345@gmail.com>> 写道：
> > >>>>>>>
> > >>>>>>> Hi Paul,
> > >>>>>>>
> > >>>>>>> +1 for this feature and supporting SQL file + JSON plans. We get
> a
> > >>>>>>>
> > >>>>>>> lot
> > >>>>>>>
> > >>>>>>> of
> > >>>>>>>
> > >>>>>>> requests to just be able to submit a SQL file, but the JSON plan
> > >>>>>>> optimizations make sense.
> > >>>>>>>
> > >>>>>>> +1 for init containers or a more generalized way of obtaining
> > >>>>>>>
> > >>>>>>> arbitrary
> > >>>>>>>
> > >>>>>>> files. File fetching isn't specific to just SQL--it also matters
> > for
> > >>>>>>>
> > >>>>>>> Java
> > >>>>>>>
> > >>>>>>> applications if the user doesn't want to rebuild a Flink image
> and
> > >>>>>>>
> > >>>>>>> just
> > >>>>>>>
> > >>>>>>> wants to modify the user application fat jar.
> > >>>>>>>
> > >>>>>>> Please note that we could reuse the checkpoint storage like
> > S3/HDFS,
> > >>>>>>>
> > >>>>>>> which
> > >>>>>>>
> > >>>>>>> should
> > >>>>>>>
> > >>>>>>> be required to run Flink in production, so I guess that would be
> > >>>>>>>
> > >>>>>>> acceptable
> > >>>>>>>
> > >>>>>>> for most
> > >>>>>>>
> > >>>>>>> users. WDYT?
> > >>>>>>>
> > >>>>>>> If you do go this route, it would be nice to support writing
> these
> > >>>>>>>
> > >>>>>>> files
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> S3/HDFS via Flink. This makes access control and policy
> management
> > >>>>>>>
> > >>>>>>> simpler.
> > >>>>>>>
> > >>>>>>> Also, what do you think about prefixing the config options with
> > >>>>>>> `sql-driver` instead of just `sql` to be more specific?
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Mason
> > >>>>>>>
> > >>>>>>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
> > <ma...@gmail.com> mailto:paullin3280@gmail.com
> > >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
> > >>>>>>>
> > >>>>>>> <mailto:
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>> wrote:
> > >>>>>>>
> > >>>>>>> Hi Jark,
> > >>>>>>>
> > >>>>>>> Thanks for your input! Please see my comments inline.
> > >>>>>>>
> > >>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink
> SQL?
> > >>>>>>> DataStream API also doesn't provide a default main class for
> users,
> > >>>>>>> why do we need to provide such one for SQL?
> > >>>>>>>
> > >>>>>>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> > >>>>>>>
> > >>>>>>> submitted
> > >>>>>>>
> > >>>>>>> via Flink CLI which actually could be DataStream/Table jobs.
> > >>>>>>>
> > >>>>>>> I think a default main class would be user-friendly which
> > eliminates
> > >>>>>>>
> > >>>>>>> the
> > >>>>>>>
> > >>>>>>> need
> > >>>>>>> for users to write a main class as SQLRunner in Flink K8s
> operator
> > >>>>>>>
> > >>>>>>> [1].
> > >>>>>>>
> > >>>>>>> I thought the proposed SqlDriver was a dedicated main class
> > >>>>>>>
> > >>>>>>> accepting
> > >>>>>>>
> > >>>>>>> SQL files, is
> > >>>>>>>
> > >>>>>>> that correct?
> > >>>>>>>
> > >>>>>>> Both JSON plans and SQL files are accepted. SQL Gateway should
> use
> > >>>>>>>
> > >>>>>>> JSON
> > >>>>>>>
> > >>>>>>> plans,
> > >>>>>>> while CLI users may use either JSON plans or SQL files.
> > >>>>>>>
> > >>>>>>> Please see the updated FLIP[2] for more details.
> > >>>>>>>
> > >>>>>>> Personally, I prefer the way of init containers which doesn't
> > >>>>>>>
> > >>>>>>> depend
> > >>>>>>>
> > >>>>>>> on
> > >>>>>>>
> > >>>>>>> additional components.
> > >>>>>>> This can reduce the moving parts of a production environment.
> > >>>>>>> Depending on a distributed file system makes the testing, demo,
> and
> > >>>>>>>
> > >>>>>>> local
> > >>>>>>>
> > >>>>>>> setup harder than init containers.
> > >>>>>>>
> > >>>>>>> Please note that we could reuse the checkpoint storage like
> > S3/HDFS,
> > >>>>>>>
> > >>>>>>> which
> > >>>>>>>
> > >>>>>>> should
> > >>>>>>> be required to run Flink in production, so I guess that would be
> > >>>>>>> acceptable for most
> > >>>>>>> users. WDYT?
> > >>>>>>>
> > >>>>>>> WRT testing, demo, and local setups, I think we could support the
> > >>>>>>>
> > >>>>>>> local
> > >>>>>>>
> > >>>>>>> filesystem
> > >>>>>>> scheme i.e. file://** <file:///**> <file:///**> <> as the state
> > backends do. It works as long as
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> Gateway
> > >>>>>>> and JobManager(or SQL Driver) can access the resource directory
> > >>>>>>>
> > >>>>>>> (specified
> > >>>>>>>
> > >>>>>>> via
> > >>>>>>> `sql-gateway.application.storage-dir`).
> > >>>>>>>
> > >>>>>>> Thanks!
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.javahttps:/*github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHSwRQyci$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.javahttps:/*github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHSwRQyci$
> > >>>>
> > >>>>>>> [2]
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:*Introduce*SQL*Driver__;Kysr!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRSnxRKF$
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:*Introduce*SQL*Driver__;Kysr!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRSnxRKF$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:*Introduce*SQL*Driver__;Kysr!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRSnxRKF$
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:*Introduce*SQL*Driver__;Kysr!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRSnxRKF$
> > >>>>
> > >>>>>>> [3]
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java*L161https:/*github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java*L161__;Iy8j!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHWwAkTvH$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java*L161https:/*github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java*L161__;Iy8j!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHWwAkTvH$
> > >>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>> 2023年6月3日 12:21，Jark Wu <imjark@gmail.com <mailto:
> imjark@gmail.com>
> > mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com
> >>
> > >>>>>>> 写道：
> > >>>>>>>
> > >>>>>>> Hi Paul,
> > >>>>>>>
> > >>>>>>> Thanks for your reply. I left my comments inline.
> > >>>>>>>
> > >>>>>>> As the FLIP said, it’s good to have a default main class for
> Flink
> > >>>>>>>
> > >>>>>>> SQLs,
> > >>>>>>>
> > >>>>>>> which allows users to submit Flink SQLs in the same way as
> > >>>>>>>
> > >>>>>>> DataStream
> > >>>>>>>
> > >>>>>>> jobs, or else users need to write their own main class.
> > >>>>>>>
> > >>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink
> SQL?
> > >>>>>>> DataStream API also doesn't provide a default main class for
> users,
> > >>>>>>> why do we need to provide such one for SQL?
> > >>>>>>>
> > >>>>>>> With the help of ExecNodeGraph, do we still need the serialized
> > >>>>>>> SessionState? If not, we could make SQL Driver accepts two
> > >>>>>>>
> > >>>>>>> serialized
> > >>>>>>>
> > >>>>>>> formats:
> > >>>>>>>
> > >>>>>>> No, ExecNodeGraph doesn't need to serialize SessionState. I
> thought
> > >>>>>>>
> > >>>>>>> the
> > >>>>>>>
> > >>>>>>> proposed SqlDriver was a dedicated main class accepting SQL
> files,
> > >>>>>>>
> > >>>>>>> is
> > >>>>>>>
> > >>>>>>> that correct?
> > >>>>>>> If true, we have to ship the SessionState for this case which is
> a
> > >>>>>>>
> > >>>>>>> large
> > >>>>>>>
> > >>>>>>> work.
> > >>>>>>> I think we just need a JsonPlanDriver which is a main class that
> > >>>>>>>
> > >>>>>>> accepts
> > >>>>>>>
> > >>>>>>> JsonPlan as the parameter.
> > >>>>>>>
> > >>>>>>> The common solutions I know is to use distributed file systems or
> > >>>>>>>
> > >>>>>>> use
> > >>>>>>>
> > >>>>>>> init containers to localize the resources.
> > >>>>>>>
> > >>>>>>> Personally, I prefer the way of init containers which doesn't
> > >>>>>>>
> > >>>>>>> depend
> > >>>>>>>
> > >>>>>>> on
> > >>>>>>>
> > >>>>>>> additional components.
> > >>>>>>> This can reduce the moving parts of a production environment.
> > >>>>>>> Depending on a distributed file system makes the testing, demo,
> and
> > >>>>>>>
> > >>>>>>> local
> > >>>>>>>
> > >>>>>>> setup harder than init containers.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Jark
> > >>>>>>>
> > >>>>>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
> > <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com>
> > >>>>>>>
> > >>>>>>> <mailto:
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>> <mailto:
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com> <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>>> wrote:
> > >>>>>>>
> > >>>>>>> The FLIP is in the early phase and some details are not included,
> > >>>>>>>
> > >>>>>>> but
> > >>>>>>>
> > >>>>>>> fortunately, we got lots of valuable ideas from the discussion.
> > >>>>>>>
> > >>>>>>> Thanks to everyone who joined the dissuasion!
> > >>>>>>> @Weihua @Shanmon @Shengkai @Biao @Jark
> > >>>>>>>
> > >>>>>>> This weekend I’m gonna revisit and update the FLIP, adding more
> > >>>>>>> details. Hopefully, we can further align our opinions.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
> > paullin3280@gmail.com> mailto:paullin3280@gmail.com <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com> <mailto:
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>> 写道：
> > >>>>>>>
> > >>>>>>> Hi Jark,
> > >>>>>>>
> > >>>>>>> Thanks a lot for your input!
> > >>>>>>>
> > >>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> > >>>>>>>
> > >>>>>>> still
> > >>>>>>>
> > >>>>>>> necessary to support SQL Driver?
> > >>>>>>>
> > >>>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
> > >>>>>>>
> > >>>>>>> simplify
> > >>>>>>>
> > >>>>>>> Flink SQL execution with Flink CLI.
> > >>>>>>>
> > >>>>>>> As the FLIP said, it’s good to have a default main class for
> > >>>>>>>
> > >>>>>>> Flink
> > >>>>>>>
> > >>>>>>> SQLs,
> > >>>>>>>
> > >>>>>>> which allows users to submit Flink SQLs in the same way as
> > >>>>>>>
> > >>>>>>> DataStream
> > >>>>>>>
> > >>>>>>> jobs, or else users need to write their own main class.
> > >>>>>>>
> > >>>>>>> SQL Driver needs to serialize SessionState which is very
> > >>>>>>>
> > >>>>>>> challenging
> > >>>>>>>
> > >>>>>>> but not detailed covered in the FLIP.
> > >>>>>>>
> > >>>>>>> With the help of ExecNodeGraph, do we still need the serialized
> > >>>>>>> SessionState? If not, we could make SQL Driver accepts two
> > >>>>>>>
> > >>>>>>> serialized
> > >>>>>>>
> > >>>>>>> formats:
> > >>>>>>>
> > >>>>>>> - SQL files for user-facing public usage
> > >>>>>>> - ExecNodeGraph for internal usage
> > >>>>>>>
> > >>>>>>> It’s kind of similar to the relationship between job jars and
> > >>>>>>>
> > >>>>>>> jobgraphs.
> > >>>>>>>
> > >>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> > >>>>>>>
> > >>>>>>> true?
> > >>>>>>>
> > >>>>>>> Is it
> > >>>>>>>
> > >>>>>>> possible to support it?
> > >>>>>>>
> > >>>>>>> Yes, K8s doesn’t distribute any files. It’s the users’
> > >>>>>>>
> > >>>>>>> responsibility
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> make
> > >>>>>>>
> > >>>>>>> sure the resources are accessible in the containers. The common
> > >>>>>>>
> > >>>>>>> solutions
> > >>>>>>>
> > >>>>>>> I know is to use distributed file systems or use init containers
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> localize the
> > >>>>>>>
> > >>>>>>> resources.
> > >>>>>>>
> > >>>>>>> Now I lean toward introducing a fs to do the distribution job.
> > >>>>>>>
> > >>>>>>> WDYT?
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
> imjark@gmail.com>
> > mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com
> >
> > >>>>>>> <mailto:
> > >>>>>>>
> > >>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:
> imjark@gmail.com
> > <mailto:imjark@gmail.com mailto:imjark@gmail.com>>
> > >>>>>>>
> > >>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com <mailto:
> > imjark@gmail.com mailto:imjark@gmail.com> <mailto:
> > >>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:
> imjark@gmail.com
> > <mailto:imjark@gmail.com mailto:imjark@gmail.com>>>
> > >>>>>>>
> > >>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com <mailto:
> > imjark@gmail.com mailto:imjark@gmail.com> <mailto:
> > >>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:
> imjark@gmail.com
> > <mailto:imjark@gmail.com mailto:imjark@gmail.com>> <mailto:
> > >>>>>>>
> > >>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:
> imjark@gmail.com
> > <mailto:imjark@gmail.com mailto:imjark@gmail.com> <mailto:
> > imjark@gmail.commailto:imjark@gmail.com
> > >>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com>>>>>
> > >>>>>>>
> > >>>>>>> 写道：
> > >>>>>>>
> > >>>>>>> Hi Paul,
> > >>>>>>>
> > >>>>>>> Thanks for starting this discussion. I like the proposal! This
> > >>>>>>>
> > >>>>>>> is
> > >>>>>>>
> > >>>>>>> a
> > >>>>>>>
> > >>>>>>> frequently requested feature!
> > >>>>>>>
> > >>>>>>> I agree with Shengkai that ExecNodeGraph as the submission
> > >>>>>>>
> > >>>>>>> object
> > >>>>>>>
> > >>>>>>> is a
> > >>>>>>>
> > >>>>>>> better idea than SQL file. To be more specific, it should be
> > >>>>>>>
> > >>>>>>> JsonPlanGraph
> > >>>>>>>
> > >>>>>>> or CompiledPlan which is the serializable representation.
> > >>>>>>>
> > >>>>>>> CompiledPlan
> > >>>>>>>
> > >>>>>>> is a
> > >>>>>>>
> > >>>>>>> clear separation between compiling/optimization/validation and
> > >>>>>>>
> > >>>>>>> execution.
> > >>>>>>>
> > >>>>>>> This can keep the validation and metadata accessing still on the
> > >>>>>>>
> > >>>>>>> SQLGateway
> > >>>>>>>
> > >>>>>>> side. This allows SQLGateway to leverage some metadata caching
> > >>>>>>>
> > >>>>>>> and
> > >>>>>>>
> > >>>>>>> UDF
> > >>>>>>>
> > >>>>>>> JAR
> > >>>>>>>
> > >>>>>>> caching for better compiling performance.
> > >>>>>>>
> > >>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> > >>>>>>>
> > >>>>>>> still
> > >>>>>>>
> > >>>>>>> necessary to support SQL Driver? Regarding non-interactive SQL
> > >>>>>>>
> > >>>>>>> jobs,
> > >>>>>>>
> > >>>>>>> users
> > >>>>>>>
> > >>>>>>> can use the Table API program for application mode. SQL Driver
> > >>>>>>>
> > >>>>>>> needs
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> serialize SessionState which is very challenging but not
> > >>>>>>>
> > >>>>>>> detailed
> > >>>>>>>
> > >>>>>>> covered
> > >>>>>>>
> > >>>>>>> in the FLIP.
> > >>>>>>>
> > >>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> > >>>>>>>
> > >>>>>>> true?
> > >>>>>>>
> > >>>>>>> Is it
> > >>>>>>>
> > >>>>>>> possible to support it?
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Jark
> > >>>>>>>
> > >>>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
> > <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com>
> > >>>>>>>
> > >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> > >>>>>>> <mailto:
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com> <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>> <mailto:
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com> <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>> <mailto:
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com> <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Hi Weihua,
> > >>>>>>>
> > >>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
> > >>>>>>>
> > >>>>>>> challenging
> > >>>>>>>
> > >>>>>>> parts of this FLIP.
> > >>>>>>>
> > >>>>>>> Web submission is not enabled in application mode currently as
> > >>>>>>>
> > >>>>>>> you
> > >>>>>>>
> > >>>>>>> said,
> > >>>>>>>
> > >>>>>>> but it could be changed if we have good reasons.
> > >>>>>>>
> > >>>>>>> What do you think about introducing a distributed storage for
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> Gateway?
> > >>>>>>>
> > >>>>>>> We could make use of Flink file systems [1] to distribute the
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> Gateway
> > >>>>>>>
> > >>>>>>> generated resources, that should solve the problem at its root
> > >>>>>>>
> > >>>>>>> cause.
> > >>>>>>>
> > >>>>>>> Users could specify Flink-supported file systems to ship files.
> > >>>>>>>
> > >>>>>>> It’s
> > >>>>>>>
> > >>>>>>> only
> > >>>>>>>
> > >>>>>>> required when using SQL Gateway with K8s application mode.
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHU8q-5n1$
> > >>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> > huweihua.ckl@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:
> > >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> > huweihua.ckl@gmail.com> <mailto:
> > >>>>>>>
> > >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> > huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> > huweihua.ckl@gmail.com>> <mailto:
> > >>>>>>>
> > >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> > huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> > huweihua.ckl@gmail.com> <mailto:
> > >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> > huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> > huweihua.ckl@gmail.com>>>> 写道：
> > >>>>>>>
> > >>>>>>> Thanks Paul for your reply.
> > >>>>>>>
> > >>>>>>> SQLDriver looks good to me.
> > >>>>>>>
> > >>>>>>> 2. Do you mean a pass the SQL string a configuration or a
> > >>>>>>>
> > >>>>>>> program
> > >>>>>>>
> > >>>>>>> argument?
> > >>>>>>>
> > >>>>>>> I brought this up because we were unable to pass the SQL file
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> Flink
> > >>>>>>>
> > >>>>>>> using Kubernetes mode.
> > >>>>>>> For DataStream/Python users, they need to prepare their images
> > >>>>>>>
> > >>>>>>> for
> > >>>>>>>
> > >>>>>>> the
> > >>>>>>>
> > >>>>>>> jars
> > >>>>>>>
> > >>>>>>> and dependencies.
> > >>>>>>> But for SQL users, they can use a common image to run
> > >>>>>>>
> > >>>>>>> different
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> queries
> > >>>>>>>
> > >>>>>>> if there are no other udf requirements.
> > >>>>>>> It would be great if the SQL query and image were not bound.
> > >>>>>>>
> > >>>>>>> Using strings is a way to decouple these, but just as you
> > >>>>>>>
> > >>>>>>> mentioned,
> > >>>>>>>
> > >>>>>>> it's
> > >>>>>>>
> > >>>>>>> not easy to pass complex SQL.
> > >>>>>>>
> > >>>>>>> use web submission
> > >>>>>>>
> > >>>>>>> AFAIK, we can not use web submission in the Application mode.
> > >>>>>>>
> > >>>>>>> Please
> > >>>>>>>
> > >>>>>>> correct me if I'm wrong.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Weihua
> > >>>>>>>
> > >>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>
> > >>>>>>>
> > >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> > >>>>>>>
> > >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>>>
> > >>>>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Hi Biao,
> > >>>>>>>
> > >>>>>>> Thanks for your comments!
> > >>>>>>>
> > >>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
> > >>>>>>>
> > >>>>>>> Flink
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> jobs
> > >>>>>>>
> > >>>>>>> in
> > >>>>>>>
> > >>>>>>> Application mode? More specifically, if we use SQL
> > >>>>>>>
> > >>>>>>> client/gateway
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> execute some interactive SQLs like a SELECT query, can we
> > >>>>>>>
> > >>>>>>> ask
> > >>>>>>>
> > >>>>>>> flink
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> use
> > >>>>>>>
> > >>>>>>> Application mode to execute those queries after this FLIP?
> > >>>>>>>
> > >>>>>>> Thanks for pointing it out. I think only DMLs would be
> > >>>>>>>
> > >>>>>>> executed
> > >>>>>>>
> > >>>>>>> via
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> Driver.
> > >>>>>>> I'll add the scope to the FLIP.
> > >>>>>>>
> > >>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> > >>>>>>>
> > >>>>>>> trivial as
> > >>>>>>>
> > >>>>>>> we
> > >>>>>>>
> > >>>>>>> can ship files via YARN's tool easily but for K8s, things
> > >>>>>>>
> > >>>>>>> can
> > >>>>>>>
> > >>>>>>> be
> > >>>>>>>
> > >>>>>>> more
> > >>>>>>>
> > >>>>>>> complicated as Shengkai said.
> > >>>>>>>
> > >>>>>>> Your input is very informative. I’m thinking about using web
> > >>>>>>>
> > >>>>>>> submission,
> > >>>>>>>
> > >>>>>>> but it requires exposing the JobManager port which could also
> > >>>>>>>
> > >>>>>>> be
> > >>>>>>>
> > >>>>>>> a
> > >>>>>>>
> > >>>>>>> problem
> > >>>>>>>
> > >>>>>>> on K8s.
> > >>>>>>>
> > >>>>>>> Another approach is to explicitly require a distributed
> > >>>>>>>
> > >>>>>>> storage
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> ship
> > >>>>>>>
> > >>>>>>> files,
> > >>>>>>> but we may need a new deployment executor for that.
> > >>>>>>>
> > >>>>>>> What do you think of these two approaches?
> > >>>>>>>
> > >>>>>>> 3. Serialization of SessionState: in SessionState, there are
> > >>>>>>>
> > >>>>>>> some
> > >>>>>>>
> > >>>>>>> unserializable fields
> > >>>>>>> like
> > >>>>>>>
> > >>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> > >>>>>>>
> > >>>>>>> It
> > >>>>>>>
> > >>>>>>> may be worthwhile to add more details about the
> > >>>>>>>
> > >>>>>>> serialization
> > >>>>>>>
> > >>>>>>> part.
> > >>>>>>>
> > >>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph
> > >>>>>>>
> > >>>>>>> as
> > >>>>>>>
> > >>>>>>> Shengkai
> > >>>>>>>
> > >>>>>>> mentioned, do we eliminate the need for serialization of
> > >>>>>>>
> > >>>>>>> SessionState?
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
> > biaogeng7@gmail.com> mailto:biaogeng7@gmail.com <mailto:
> > >>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:
> > biaogeng7@gmail.com> <mailto:
> > >>>>>>>
> > >>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:
> > biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:
> biaogeng7@gmail.com>>
> > <mailto:
> > >>>>>>>
> > >>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:
> > biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:
> biaogeng7@gmail.com>
> > <mailto:
> > >>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:
> > biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:
> biaogeng7@gmail.com>>>>
> > 写道：
> > >>>>>>>
> > >>>>>>> Thanks Paul for the proposal!I believe it would be very
> > >>>>>>>
> > >>>>>>> useful
> > >>>>>>>
> > >>>>>>> for
> > >>>>>>>
> > >>>>>>> flink
> > >>>>>>>
> > >>>>>>> users.
> > >>>>>>> After reading the FLIP, I have some questions:
> > >>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
> > >>>>>>>
> > >>>>>>> Flink
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> jobs
> > >>>>>>>
> > >>>>>>> in
> > >>>>>>>
> > >>>>>>> Application mode? More specifically, if we use SQL
> > >>>>>>>
> > >>>>>>> client/gateway
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> execute some interactive SQLs like a SELECT query, can we
> > >>>>>>>
> > >>>>>>> ask
> > >>>>>>>
> > >>>>>>> flink
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> use
> > >>>>>>>
> > >>>>>>> Application mode to execute those queries after this FLIP?
> > >>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> > >>>>>>>
> > >>>>>>> trivial as
> > >>>>>>>
> > >>>>>>> we
> > >>>>>>>
> > >>>>>>> can ship files via YARN's tool easily but for K8s, things
> > >>>>>>>
> > >>>>>>> can
> > >>>>>>>
> > >>>>>>> be
> > >>>>>>>
> > >>>>>>> more
> > >>>>>>>
> > >>>>>>> complicated as Shengkai said. I have implemented a simple
> > >>>>>>>
> > >>>>>>> POC
> > >>>>>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https:/*github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHalMKQ1M$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https:/*github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHalMKQ1M$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https:/*github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHalMKQ1M$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https:/*github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHalMKQ1M$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https:/*github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHalMKQ1M$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https:/*github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHalMKQ1M$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https:/*github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHalMKQ1M$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https:/*github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHalMKQ1M$
> > >>>>
> > >>>>>>> based on SQL client before(i.e. consider the SQL client
> > >>>>>>>
> > >>>>>>> which
> > >>>>>>>
> > >>>>>>> supports
> > >>>>>>>
> > >>>>>>> executing a SQL file as the SQL driver in this FLIP). One
> > >>>>>>>
> > >>>>>>> problem
> > >>>>>>>
> > >>>>>>> I
> > >>>>>>>
> > >>>>>>> have
> > >>>>>>>
> > >>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s
> > >>>>>>>
> > >>>>>>> side.
> > >>>>>>>
> > >>>>>>> Without
> > >>>>>>>
> > >>>>>>> such support, users have to modify the initContainer or
> > >>>>>>>
> > >>>>>>> rebuild
> > >>>>>>>
> > >>>>>>> a
> > >>>>>>>
> > >>>>>>> new
> > >>>>>>>
> > >>>>>>> K8s
> > >>>>>>>
> > >>>>>>> image every time to fetch the SQL file. Like the flink k8s
> > >>>>>>>
> > >>>>>>> operator,
> > >>>>>>>
> > >>>>>>> one
> > >>>>>>>
> > >>>>>>> workaround is to utilize the flink config(transforming the
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> file
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> a
> > >>>>>>>
> > >>>>>>> escaped string like Weihua mentioned) which will be
> > >>>>>>>
> > >>>>>>> converted
> > >>>>>>>
> > >>>>>>> to a
> > >>>>>>>
> > >>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger
> > >>>>>>>
> > >>>>>>> than
> > >>>>>>>
> > >>>>>>> 1MB
> > >>>>>>>
> > >>>>>>> <
> > >>>>>>>
> > >>>>>>>
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> <
> > >>>>>>>
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> > >>>>>>>
> > >>>>>>> <
> > >>>>>>>
> > >>>>>>>
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> <
> > >>>>>>>
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >> <
> > >>>>>>>
> > >>>>>>>
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> <
> > >>>>>>>
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> > <
> > >>>>>>>
> > >>>>>>>
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> <
> > >>>>>>>
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >
> https://urldefense.com/v3/__https://kubernetes.io/docs/concepts/configuration/configmap/__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHQpdIxub$
> >>>>).
> > >>>>>>>
> > >>>>>>> Not
> > >>>>>>>
> > >>>>>>> sure
> > >>>>>>>
> > >>>>>>> if we have better solutions.
> > >>>>>>> 3. Serialization of SessionState: in SessionState, there are
> > >>>>>>>
> > >>>>>>> some
> > >>>>>>>
> > >>>>>>> unserializable fields
> > >>>>>>> like
> > >>>>>>>
> > >>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> > >>>>>>>
> > >>>>>>> It
> > >>>>>>>
> > >>>>>>> may be worthwhile to add more details about the
> > >>>>>>>
> > >>>>>>> serialization
> > >>>>>>>
> > >>>>>>> part.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Biao Geng
> > >>>>>>>
> > >>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> > mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>
> > >>>>>>> <mailto:
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>
> > >>>>>>>
> > >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>>>
> > >>>>>>>
> > >>>>>>> 于2023年5月31日周三 11:49写道：
> > >>>>>>>
> > >>>>>>> Hi Weihua,
> > >>>>>>>
> > >>>>>>> Thanks a lot for your input! Please see my comments inline.
> > >>>>>>>
> > >>>>>>> - Is SQLRunner the better name? We use this to run a SQL
> > >>>>>>>
> > >>>>>>> Job.
> > >>>>>>>
> > >>>>>>> (Not
> > >>>>>>>
> > >>>>>>> strong,
> > >>>>>>>
> > >>>>>>> the SQLDriver is fine for me)
> > >>>>>>>
> > >>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
> > >>>>>>>
> > >>>>>>> following
> > >>>>>>>
> > >>>>>>> reasons FYI:
> > >>>>>>>
> > >>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
> > >>>>>>> 2. Flink program's main class is sort of like Driver in
> > >>>>>>>
> > >>>>>>> JDBC
> > >>>>>>>
> > >>>>>>> which
> > >>>>>>>
> > >>>>>>> translates SQLs into
> > >>>>>>> databases specific languages.
> > >>>>>>>
> > >>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> > >>>>>>>
> > >>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
> > >>>>>>>
> > >>>>>>> we
> > >>>>>>>
> > >>>>>>> need
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> prepare
> > >>>>>>>
> > >>>>>>> a SQL file in an image for Kubernetes application mode,
> > >>>>>>>
> > >>>>>>> which
> > >>>>>>>
> > >>>>>>> may
> > >>>>>>>
> > >>>>>>> be
> > >>>>>>>
> > >>>>>>> a
> > >>>>>>>
> > >>>>>>> bit
> > >>>>>>>
> > >>>>>>> cumbersome.
> > >>>>>>>
> > >>>>>>> Do you mean a pass the SQL string a configuration or a
> > >>>>>>>
> > >>>>>>> program
> > >>>>>>>
> > >>>>>>> argument?
> > >>>>>>>
> > >>>>>>> I thought it might be convenient for testing propose, but
> > >>>>>>>
> > >>>>>>> not
> > >>>>>>>
> > >>>>>>> recommended
> > >>>>>>>
> > >>>>>>> for production,
> > >>>>>>> cause Flink SQLs could be complicated and involves lots of
> > >>>>>>>
> > >>>>>>> characters
> > >>>>>>>
> > >>>>>>> that
> > >>>>>>>
> > >>>>>>> need to escape.
> > >>>>>>>
> > >>>>>>> WDYT?
> > >>>>>>>
> > >>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> > >>>>>>>
> > >>>>>>> "run-application"
> > >>>>>>>
> > >>>>>>> command. Does that mean we need to perform automatic
> > >>>>>>>
> > >>>>>>> detection
> > >>>>>>>
> > >>>>>>> in
> > >>>>>>>
> > >>>>>>> Flink?
> > >>>>>>>
> > >>>>>>> Yes! It’s like running a PyFlink job with the following
> > >>>>>>>
> > >>>>>>> command:
> > >>>>>>>
> > >>>>>>> `./bin/flink run \\ --pyModule table.word_count \\ --pyFiles
> > examples/python/table`
> > >>>>>>>
> > >>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
> > >>>>>>>
> > >>>>>>> Driver
> > >>>>>>>
> > >>>>>>> automatically.
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps:/*github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTnj4MLa$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps:/*github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTnj4MLa$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps:/*github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTnj4MLa$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps:/*github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTnj4MLa$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps:/*github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTnj4MLa$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps:/*github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTnj4MLa$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps:/*github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTnj4MLa$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps:/*github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java__;Lw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTnj4MLa$
> > >>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> > huweihua.ckl@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:
> > >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> > huweihua.ckl@gmail.com>
> > >>>>>>>
> > >>>>>>> <mailto:
> > >>>>>>>
> > >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> > huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> > huweihua.ckl@gmail.com>> <mailto:
> > >>>>>>>
> > >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> > huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> > huweihua.ckl@gmail.com>>> 写道：
> > >>>>>>>
> > >>>>>>> Thanks Paul for the proposal.
> > >>>>>>>
> > >>>>>>> +1 for this. It is valuable in improving ease of use.
> > >>>>>>>
> > >>>>>>> I have a few questions.
> > >>>>>>> - Is SQLRunner the better name? We use this to run a SQL
> > >>>>>>>
> > >>>>>>> Job.
> > >>>>>>>
> > >>>>>>> (Not
> > >>>>>>>
> > >>>>>>> strong,
> > >>>>>>>
> > >>>>>>> the SQLDriver is fine for me)
> > >>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
> > >>>>>>>
> > >>>>>>> we
> > >>>>>>>
> > >>>>>>> need
> > >>>>>>>
> > >>>>>>> to
> > >>>>>>>
> > >>>>>>> prepare
> > >>>>>>>
> > >>>>>>> a SQL file in an image for Kubernetes application mode,
> > >>>>>>>
> > >>>>>>> which
> > >>>>>>>
> > >>>>>>> may
> > >>>>>>>
> > >>>>>>> be
> > >>>>>>>
> > >>>>>>> a
> > >>>>>>>
> > >>>>>>> bit
> > >>>>>>>
> > >>>>>>> cumbersome.
> > >>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> > >>>>>>>
> > >>>>>>> "run-application"
> > >>>>>>>
> > >>>>>>> command. Does that mean we need to perform automatic
> > >>>>>>>
> > >>>>>>> detection
> > >>>>>>>
> > >>>>>>> in
> > >>>>>>>
> > >>>>>>> Flink?
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Weihua
> > >>>>>>>
> > >>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> > >>>>>>>
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>
> > >>>>>>>
> > >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> > paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> > paullin3280@gmail.com>>>>
> > >>>>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Hi team,
> > >>>>>>>
> > >>>>>>> I’d like to start a discussion about FLIP-316 [1], which
> > >>>>>>>
> > >>>>>>> introduces
> > >>>>>>>
> > >>>>>>> a
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> driver as the
> > >>>>>>> default main class for Flink SQL jobs.
> > >>>>>>>
> > >>>>>>> Currently, Flink SQL could be executed out of the box
> > >>>>>>>
> > >>>>>>> either
> > >>>>>>>
> > >>>>>>> via
> > >>>>>>>
> > >>>>>>> SQL
> > >>>>>>>
> > >>>>>>> Client/Gateway
> > >>>>>>> or embedded in a Flink Java/Python program.
> > >>>>>>>
> > >>>>>>> However, each one has its drawback:
> > >>>>>>>
> > >>>>>>> - SQL Client/Gateway doesn’t support the application
> > >>>>>>>
> > >>>>>>> deployment
> > >>>>>>>
> > >>>>>>> mode
> > >>>>>>>
> > >>>>>>> [2]
> > >>>>>>>
> > >>>>>>> - Flink Java/Python program requires extra work to write
> > >>>>>>>
> > >>>>>>> a
> > >>>>>>>
> > >>>>>>> non-SQL
> > >>>>>>>
> > >>>>>>> program
> > >>>>>>>
> > >>>>>>> Therefore, I propose adding a SQL driver to act as the
> > >>>>>>>
> > >>>>>>> default
> > >>>>>>>
> > >>>>>>> main
> > >>>>>>>
> > >>>>>>> class
> > >>>>>>>
> > >>>>>>> for SQL jobs.
> > >>>>>>> Please see the FLIP docs for details and feel free to
> > >>>>>>>
> > >>>>>>> comment.
> > >>>>>>>
> > >>>>>>> Thanks!
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316*3A*Introduce*SQL*Driver__;JSsrKw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTACBQ5Y$
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316*3A*Introduce*SQL*Driver__;JSsrKw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTACBQ5Y$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316*3A*Introduce*SQL*Driver__;JSsrKw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTACBQ5Y$
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316*3A*Introduce*SQL*Driver__;JSsrKw!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHTACBQ5Y$
> > >>>>
> > >>>>>>> <
> > >>>>
> > >>>>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:*Introduce*SQL*Driver__;Kysr!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRSnxRKF$
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:*Introduce*SQL*Driver__;Kysr!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRSnxRKF$
> > >>>> <
> > >>>>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:*Introduce*SQL*Driver__;Kysr!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRSnxRKF$
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:*Introduce*SQL*Driver__;Kysr!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHRSnxRKF$
> > >>>>
> > >>>>>>> [2]
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-26541__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHdHUioYM$
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-26541__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHdHUioYM$
> <
> > >>>>>>>
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-26541__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHdHUioYM$
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-26541__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHdHUioYM$
> > <
> > >>>>>>>
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-26541__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHdHUioYM$
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-26541__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHdHUioYM$
> <
> > >>>>>>>
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-26541__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHdHUioYM$
> >
> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/FLINK-26541__;!!IKRxdwAv5BmarQ!fIUTu2bEolYza9fivJqHWl5aib42VbcVbRDyF0cdiYQ8IwAPYYFQO-YPvVOC4q4tlJZu4Tat11e0mmFkHdHUioYM$
> >>
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> >
> >
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Márton Balassi <ba...@gmail.com>.
Thanks, Paul.
Ferenc and I have been looking into unblocking the Kubernetes path via an
updated implementation for FLINK-28915 to ship the jars conveniently there.
You can expect an updated PR there next week. Looking forward to your
findings in the YARN POC.
On Mon, Dec 11, 2023 at 4:01 AM Paul Lam <pa...@gmail.com> wrote:
> Hi Ferenc,
>
> Sorry for my late reply.
>
> > Is any active work happening on this FLIP? As far as I see there
> > are blockers that needs to happen first to implement regarding
> > artifact distribution.
>
> You’re right. There’s a block in K8s application mode, but none in
> YARN application. I’m doing a POC on YARN application mode
> before starting a vote thread.
>
> I’ve been busy lately, but the FLIP is active for sure. The situation
> will change in a couple of weeks.
>
> Thank you for reaching out! I’ll let you know when the POC is completed.
>
> Best,
> Paul Lam
>
> > 2023年11月21日 06:01，Ferenc Csaky <fe...@pm.me.INVALID> 写道：
> >
> > Hello devs,
> >
> > Is any active work happening on this FLIP? As far as I see there
> > are blockers that needs to happen first to implement regarding
> > artifact distribution.
> >
> > Is this work in halt completetly or some efforts are going into
> > resolve the blockers first or something?
> >
> > Our platform would benefit this feature a lot, we have a kind of
> > working custom implementation at the moment, but it is uniquely
> > adapted to our app and platform.
> >
> > I could help out to move this forward.
> >
> > Best,
> > Ferenc
> >
> >
> >
> > On Friday, June 30th, 2023 at 04:53, Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >
> >>
> >>
> >> Hi Jing,
> >>
> >> Thanks for your input!
> >>
> >>> Would you like to add
> >>> one section to describe(better with script/code example) how to use it
> in
> >>> these two scenarios from users' perspective?
> >>
> >>
> >> OK. I’ll update the FLIP with the code snippet after I get the POC
> branch done.
> >>
> >>> NIT: the pictures have transparent background when readers click on
> it. It
> >>> would be great if you can replace them with pictures with white
> background.
> >>
> >>
> >> Fixed. Thanks for pointing that out :)
> >>
> >> Best,
> >> Paul Lam
> >>
> >>> 2023年6月27日 06:51，Jing Ge jing@ververica.com.INVALID 写道：
> >>>
> >>> Hi Paul,
> >>>
> >>> Thanks for driving it and thank you all for the informative
> discussion! The
> >>> FLIP is in good shape now. As described in the FLIP, SQL Driver will be
> >>> mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway
> in
> >>> application mode and 2. external system integration. Would you like to
> add
> >>> one section to describe(better with script/code example) how to use it
> in
> >>> these two scenarios from users' perspective?
> >>>
> >>> NIT: the pictures have transparent background when readers click on
> it. It
> >>> would be great if you can replace them with pictures with white
> background.
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>> On Mon, Jun 26, 2023 at 1:31 PM Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com> mailto:paullin3280@gmail.com> wrote:
> >>>
> >>>> Hi Shengkai,
> >>>>
> >>>>> * How can we ship the json plan to the JobManager?
> >>>>
> >>>> The Flink K8s module should be responsible for file distribution. We
> could
> >>>> introduce
> >>>> an option like `kubernetes.storage.dir`. For each flink cluster, there
> >>>> would be a
> >>>> dedicated subdirectory, with the pattern like
> >>>> `${kubernetes.storage.dir}/${cluster-id}`.
> >>>>
> >>>> All resources-related options (e.g. pipeline jars, json plans) that
> are
> >>>> configured with
> >>>> scheme `file://` <file://`/> <file://`/> <file:///%60 <file:///%60>>
> would be uploaded to the resource directory
> >>>> and downloaded to the
> >>>> jobmanager, before SQL Driver accesses the files with the original
> >>>> filenames.
> >>>>
> >>>>> * Classloading strategy
> >>>>
> >>>> We could directly specify the SQL Gateway jar as the jar file in
> >>>> PackagedProgram.
> >>>> It would be treated like a normal user jar and the SQL Driver is
> loaded
> >>>> into the user
> >>>> classloader. WDYT?
> >>>>
> >>>>> * Option `$internal.sql-gateway.driver.sql-config` is string type
> >>>>> I think it's better to use Map type here
> >>>>
> >>>> By Map type configuration, do you mean a nested map that contains all
> >>>> configurations?
> >>>>
> >>>> I hope I've explained myself well, it’s a file that contains the
> extra SQL
> >>>> configurations, which would be shipped to the jobmanager.
> >>>>
> >>>>> * PoC branch
> >>>>
> >>>> Sure. I’ll let you know once I get the job done.
> >>>>
> >>>> Best,
> >>>> Paul Lam
> >>>>
> >>>>> 2023年6月26日 14:27，Shengkai Fang <fskmine@gmail.com <mailto:
> fskmine@gmail.com> mailto:fskmine@gmail.com> 写道：
> >>>>>
> >>>>> Hi, Paul.
> >>>>>
> >>>>> Thanks for your update. I have a few questions about the new design:
> >>>>>
> >>>>> * How can we ship the json plan to the JobManager?
> >>>>>
> >>>>> The current design only exposes an option about the URL of the json
> >>>>> plan. It seems the gateway is responsible to upload to an external
> stroage.
> >>>>> Can we reuse the PipelineOptions.JARS to ship to the remote
> filesystem?
> >>>>>
> >>>>> * Classloading strategy
> >>>>>
> >>>>> Currently, the Driver is in the sql-gateway package. It means the
> Driver
> >>>>> is not in the JM's classpath directly. Because the sql-gateway jar
> is now
> >>>>> in the opt directory rather than lib directory. It may need to add
> the
> >>>>> external dependencies as Python does[1]. BTW, I think it's better to
> move
> >>>>> the Driver into the flink-table-runtime package, which is much
> easier to
> >>>>> find(Sorry for the wrong opinion before).
> >>>>>
> >>>>> * Option `$internal.sql-gateway.driver.sql-config` is string type
> >>>>>
> >>>>> I think it's better to use Map type here
> >>>>>
> >>>>> * PoC branch
> >>>>>
> >>>>> Because this FLIP involves many modules, do you have a PoC branch to
> >>>>> verify it does work?
> >>>>>
> >>>>> Best,
> >>>>> Shengkai
> >>>>>
> >>>>> [1]
> >>>>>
> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
> >>>>> <
> >>>>>
> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
> >>>>>
> >>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>
> >>>>> 于2023年6月19日周一 14:09写道：
> >>>>> Hi Shengkai,
> >>>>>
> >>>>> Sorry for my late reply. It took me some time to update the FLIP.
> >>>>>
> >>>>> In the latest FLIP design, SQL Driver is placed in flink-sql-gateway
> >>>>> module. PTAL.
> >>>>>
> >>>>> The FLIP does not cover details about the K8s file distribution, but
> its
> >>>>> general usage would
> >>>>> be very much the same as YARN setups. We could make follow-up
> >>>>> discussions in the jira
> >>>>> tickets.
> >>>>>
> >>>>> Best,
> >>>>> Paul Lam
> >>>>>
> >>>>>> 2023年6月12日 15:29，Shengkai Fang <fskmine@gmail.com <mailto:
> fskmine@gmail.com> mailto:fskmine@gmail.com <mailto:
> >>>>>> fskmine@gmail.com <ma...@gmail.com> mailto:
> fskmine@gmail.com>> 写道：
> >>>>>>
> >>>>>>> If it’s the case, I’m good with introducing a new module and making
> >>>>>>> SQL Driver
> >>>>>>> an internal class and accepts JSON plans only.
> >>>>>>
> >>>>>> I rethink this again and again. I think it's better to move the
> >>>>>> SqlDriver into the sql-gateway module because the sql client relies
> on the
> >>>>>> sql-gateway to submit the sql and the sql-gateway has the ability to
> >>>>>> generate the ExecNodeGraph now. +1 to support accepting JSON plans
> only.
> >>>>>>
> >>>>>> * Upload configuration through command line parameter
> >>>>>>
> >>>>>> ExecNodeGraph only contains the job's information but it doesn't
> >>>>>> contain the checkpoint dir, checkpoint interval, execution mode and
> so on.
> >>>>>> So I think we should also upload the configuration.
> >>>>>>
> >>>>>> * KubernetesClusterDescripter and
> >>>>>> KubernetesApplicationClusterEntrypoint are responsible for the jar
> >>>>>> upload/download
> >>>>>>
> >>>>>> +1 for the change.
> >>>>>>
> >>>>>> Could you update the FLIP about the current discussion?
> >>>>>>
> >>>>>> Best,
> >>>>>> Shengkai
> >>>>>>
> >>>>>> Yang Wang <wangyang0918@apache.org <ma...@apache.org>
> mailto:wangyang0918@apache.org <mailto:wangyang0918@apache.orgmailto:
> wangyang0918@apache.org>>
> >>>>>> 于2023年6月12日周一 11:41写道：
> >>>>>> Sorry for the late reply. I am in favor of introducing such a
> built-in
> >>>>>> resource localization mechanism
> >>>>>> based on Flink FileSystem. Then FLINK-28915[1] could be the second
> step
> >>>>>> which will download
> >>>>>> the jars and dependencies to the JobManager/TaskManager local
> directory
> >>>>>> before working.
> >>>>>>
> >>>>>> The first step could be done in another ticket in Flink. Or some
> >>>>>> external
> >>>>>> Flink jobs management system
> >>>>>> could also take care of this.
> >>>>>>
> >>>>>> [1]. https://issues.apache.org/jira/browse/FLINK-28915
> https://issues.apache.org/jira/browse/FLINK-28915 <
> >>>>>> https://issues.apache.org/jira/browse/FLINK-28915
> https://issues.apache.org/jira/browse/FLINK-28915>
> >>>>>>
> >>>>>> Best,
> >>>>>> Yang
> >>>>>>
> >>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>
> >>>>>> 于2023年6月9日周五 17:39写道：
> >>>>>>
> >>>>>>> Hi Mason,
> >>>>>>>
> >>>>>>> I get your point. I'm increasingly feeling the need to introduce a
> >>>>>>> built-in
> >>>>>>> file distribution mechanism for flink-kubernetes module, just like
> >>>>>>> Spark
> >>>>>>> does with `spark.kubernetes.file.upload.path` [1].
> >>>>>>>
> >>>>>>> I’m assuming the workflow is as follows:
> >>>>>>>
> >>>>>>> - KubernetesClusterDescripter uploads all local resources to a
> remote
> >>>>>>> storage via Flink filesystem (skips if the resources are already
> >>>>>>> remote).
> >>>>>>> - KubernetesApplicationClusterEntrypoint downloads the resources
> >>>>>>> and put them in the classpath during startup.
> >>>>>>>
> >>>>>>> I wouldn't mind splitting it into another FLIP to ensure that
> >>>>>>> everything is
> >>>>>>> done correctly.
> >>>>>>>
> >>>>>>> cc'ed @Yang to gather more opinions.
> >>>>>>>
> >>>>>>> [1]
> >>>>
> >>>>
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> >>>> <
> >>>>
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> >>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>> 2023年6月8日 12:15，Mason Chen <mas.chen6345@gmail.com <mailto:
> mas.chen6345@gmail.com> mailto:mas.chen6345@gmail.com <mailto:
> >>>>>>> mas.chen6345@gmail.com <ma...@gmail.com> mailto:
> mas.chen6345@gmail.com>> 写道：
> >>>>>>>
> >>>>>>> Hi Paul,
> >>>>>>>
> >>>>>>> Thanks for your response!
> >>>>>>>
> >>>>>>> I agree that utilizing SQL Drivers in Java applications is equally
> >>>>>>> important
> >>>>>>>
> >>>>>>> as employing them in SQL Gateway. WRT init containers, I think most
> >>>>>>> users use them just as a workaround. For example, wget a jar from
> the
> >>>>>>> maven repo.
> >>>>>>>
> >>>>>>> We could implement the functionality in SQL Driver in a more
> graceful
> >>>>>>> way and the flink-supported filesystem approach seems to be a
> >>>>>>> good choice.
> >>>>>>>
> >>>>>>> My main point is: can we solve the problem with a design agnostic
> of
> >>>>>>> SQL
> >>>>>>> and Stream API? I mentioned a use case where this ability is useful
> >>>>>>> for
> >>>>>>> Java or Stream API applications. Maybe this is even a non-goal to
> >>>>>>> your FLIP
> >>>>>>> since you are focusing on the driver entrypoint.
> >>>>>>>
> >>>>>>> Jark mentioned some optimizations:
> >>>>>>>
> >>>>>>> This allows SQLGateway to leverage some metadata caching and UDF
> JAR
> >>>>>>> caching for better compiling performance.
> >>>>>>>
> >>>>>>> It would be great to see this even outside the SQLGateway (i.e. UDF
> >>>>>>> JAR
> >>>>>>> caching).
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Mason
> >>>>>>>
> >>>>>>> On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskmine@gmail.com
> <ma...@gmail.com> mailto:fskmine@gmail.com
> >>>>>>> <mailto:fskmine@gmail.com mailto:fskmine@gmail.com>> wrote:
> >>>>>>>
> >>>>>>> Hi. Paul. Thanks for your update and the update makes me understand
> >>>>>>> the
> >>>>>>> design much better.
> >>>>>>>
> >>>>>>> But I still have some questions about the FLIP.
> >>>>>>>
> >>>>>>> For SQL Gateway, only DMLs need to be delegated to the SQL server
> >>>>>>> Driver. I would think about the details and update the FLIP. Do you
> >>>>>>> have
> >>>>>>>
> >>>>>>> some
> >>>>>>>
> >>>>>>> ideas already?
> >>>>>>>
> >>>>>>> If the applicaiton mode can not support library mode, I think we
> >>>>>>> should
> >>>>>>> only execute INSERT INTO and UPDATE/ DELETE statement in the
> >>>>>>> application
> >>>>>>> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> >>>>>>> statements. The ANALYZE TABLE syntax need to register the statistic
> >>>>>>> to the
> >>>>>>> catalog after job finishes and the CALL PROCEDURE statement doesn't
> >>>>>>> generate the ExecNodeGraph.
> >>>>>>>
> >>>>>>> * Introduce storage via option
> `sql-gateway.application.storage-dir`
> >>>>>>>
> >>>>>>> If we can not support to submit the jars through web submission,
> +1 to
> >>>>>>> introduce the options to upload the files. While I think the
> uploader
> >>>>>>> should be responsible to remove the uploaded jars. Can we remove
> the
> >>>>>>> jars
> >>>>>>> if the job is running or gateway exits?
> >>>>>>>
> >>>>>>> * JobID is not avaliable
> >>>>>>>
> >>>>>>> Can we use the returned rest client by ApplicationDeployer to query
> >>>>>>> the job
> >>>>>>> id? I am concerned that users don't know which job is related to
> the
> >>>>>>> submitted SQL.
> >>>>>>>
> >>>>>>> * Do we need to introduce a new module named
> flink-table-sql-runner?
> >>>>>>>
> >>>>>>> It seems we need to introduce a new module. Will the new module is
> >>>>>>> available in the distribution package? I agree with Jark that we
> >>>>>>> don't need
> >>>>>>> to introduce this for table-API users and these users have their
> main
> >>>>>>> class. If we want to make users write the k8s operator more
> easily, I
> >>>>>>> think
> >>>>>>> we should modify the k8s operator repo. If we don't need to support
> >>>>>>> SQL
> >>>>>>> files, can we make this jar only visible in the sql-gateway like we
> >>>>>>> do in
> >>>>>>> the planner loader?[1]
> >>>>>>>
> >>>>>>> [1]
> >>>>
> >>>>
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> >>>> <
> >>>>
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> >>>>
> >>>>>>> Best,
> >>>>>>> Shengkai
> >>>>>>>
> >>>>>>> Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com>
> mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> huweihua.ckl@gmail.com>>
> >>>>>>> 于2023年6月7日周三 10:52写道：
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> Thanks for updating the FLIP.
> >>>>>>>
> >>>>>>> I have two cents on the distribution of SQLs and resources.
> >>>>>>> 1. Should we support a common file distribution mechanism for k8s
> >>>>>>> application mode?
> >>>>>>> I have seen some issues and requirements on the mailing list.
> >>>>>>> In our production environment, we implement the download command in
> >>>>>>> the
> >>>>>>> CliFrontend.
> >>>>>>> And automatically add an init container to the POD for file
> >>>>>>>
> >>>>>>> downloading.
> >>>>>>>
> >>>>>>> The advantage of this
> >>>>>>> is that we can use all Flink-supported file systems to store files.
> >>>>>>>
> >>>>>>> This need more discussion. I would appreciate hearing more
> opinions.
> >>>>>>>
> >>>>>>> 2. In this FLIP, we distribute files in two different ways in YARN
> and
> >>>>>>> Kubernetes. Can we combine it in one way?
> >>>>>>> If we don't want to implement a common file distribution for k8s
> >>>>>>> application mode. Could we use the SQLDriver
> >>>>>>> to download the files both in YARN and K8S? IMO, this can reduce
> the
> >>>>>>>
> >>>>>>> cost
> >>>>>>>
> >>>>>>> of code maintenance.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Weihua
> >>>>>>>
> >>>>>>> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com> mailto:paullin3280@gmail.com
> >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> wrote:
> >>>>>>>
> >>>>>>> Hi Mason,
> >>>>>>>
> >>>>>>> Thanks for your input!
> >>>>>>>
> >>>>>>> +1 for init containers or a more generalized way of obtaining
> >>>>>>>
> >>>>>>> arbitrary
> >>>>>>>
> >>>>>>> files. File fetching isn't specific to just SQL--it also matters
> for
> >>>>>>>
> >>>>>>> Java
> >>>>>>>
> >>>>>>> applications if the user doesn't want to rebuild a Flink image and
> >>>>>>>
> >>>>>>> just
> >>>>>>>
> >>>>>>> wants to modify the user application fat jar.
> >>>>>>>
> >>>>>>> I agree that utilizing SQL Drivers in Java applications is equally
> >>>>>>> important
> >>>>>>> as employing them in SQL Gateway. WRT init containers, I think most
> >>>>>>> users use them just as a workaround. For example, wget a jar from
> the
> >>>>>>> maven repo.
> >>>>>>>
> >>>>>>> We could implement the functionality in SQL Driver in a more
> graceful
> >>>>>>> way and the flink-supported filesystem approach seems to be a
> >>>>>>> good choice.
> >>>>>>>
> >>>>>>> Also, what do you think about prefixing the config options with
> >>>>>>> `sql-driver` instead of just `sql` to be more specific?
> >>>>>>>
> >>>>>>> LGTM, since SQL Driver is a public interface and the options are
> >>>>>>> specific to it.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>> 2023年6月6日 06:30，Mason Chen <mas.chen6345@gmail.com <mailto:
> mas.chen6345@gmail.com> mailto:mas.chen6345@gmail.com <mailto:
> >>>>>>> mas.chen6345@gmail.com <ma...@gmail.com> mailto:
> mas.chen6345@gmail.com>> 写道：
> >>>>>>>
> >>>>>>> Hi Paul,
> >>>>>>>
> >>>>>>> +1 for this feature and supporting SQL file + JSON plans. We get a
> >>>>>>>
> >>>>>>> lot
> >>>>>>>
> >>>>>>> of
> >>>>>>>
> >>>>>>> requests to just be able to submit a SQL file, but the JSON plan
> >>>>>>> optimizations make sense.
> >>>>>>>
> >>>>>>> +1 for init containers or a more generalized way of obtaining
> >>>>>>>
> >>>>>>> arbitrary
> >>>>>>>
> >>>>>>> files. File fetching isn't specific to just SQL--it also matters
> for
> >>>>>>>
> >>>>>>> Java
> >>>>>>>
> >>>>>>> applications if the user doesn't want to rebuild a Flink image and
> >>>>>>>
> >>>>>>> just
> >>>>>>>
> >>>>>>> wants to modify the user application fat jar.
> >>>>>>>
> >>>>>>> Please note that we could reuse the checkpoint storage like
> S3/HDFS,
> >>>>>>>
> >>>>>>> which
> >>>>>>>
> >>>>>>> should
> >>>>>>>
> >>>>>>> be required to run Flink in production, so I guess that would be
> >>>>>>>
> >>>>>>> acceptable
> >>>>>>>
> >>>>>>> for most
> >>>>>>>
> >>>>>>> users. WDYT?
> >>>>>>>
> >>>>>>> If you do go this route, it would be nice to support writing these
> >>>>>>>
> >>>>>>> files
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> S3/HDFS via Flink. This makes access control and policy management
> >>>>>>>
> >>>>>>> simpler.
> >>>>>>>
> >>>>>>> Also, what do you think about prefixing the config options with
> >>>>>>> `sql-driver` instead of just `sql` to be more specific?
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Mason
> >>>>>>>
> >>>>>>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com> mailto:paullin3280@gmail.com
> >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
> >>>>>>>
> >>>>>>> <mailto:
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>> wrote:
> >>>>>>>
> >>>>>>> Hi Jark,
> >>>>>>>
> >>>>>>> Thanks for your input! Please see my comments inline.
> >>>>>>>
> >>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> >>>>>>> DataStream API also doesn't provide a default main class for users,
> >>>>>>> why do we need to provide such one for SQL?
> >>>>>>>
> >>>>>>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> >>>>>>>
> >>>>>>> submitted
> >>>>>>>
> >>>>>>> via Flink CLI which actually could be DataStream/Table jobs.
> >>>>>>>
> >>>>>>> I think a default main class would be user-friendly which
> eliminates
> >>>>>>>
> >>>>>>> the
> >>>>>>>
> >>>>>>> need
> >>>>>>> for users to write a main class as SQLRunner in Flink K8s operator
> >>>>>>>
> >>>>>>> [1].
> >>>>>>>
> >>>>>>> I thought the proposed SqlDriver was a dedicated main class
> >>>>>>>
> >>>>>>> accepting
> >>>>>>>
> >>>>>>> SQL files, is
> >>>>>>>
> >>>>>>> that correct?
> >>>>>>>
> >>>>>>> Both JSON plans and SQL files are accepted. SQL Gateway should use
> >>>>>>>
> >>>>>>> JSON
> >>>>>>>
> >>>>>>> plans,
> >>>>>>> while CLI users may use either JSON plans or SQL files.
> >>>>>>>
> >>>>>>> Please see the updated FLIP[2] for more details.
> >>>>>>>
> >>>>>>> Personally, I prefer the way of init containers which doesn't
> >>>>>>>
> >>>>>>> depend
> >>>>>>>
> >>>>>>> on
> >>>>>>>
> >>>>>>> additional components.
> >>>>>>> This can reduce the moving parts of a production environment.
> >>>>>>> Depending on a distributed file system makes the testing, demo, and
> >>>>>>>
> >>>>>>> local
> >>>>>>>
> >>>>>>> setup harder than init containers.
> >>>>>>>
> >>>>>>> Please note that we could reuse the checkpoint storage like
> S3/HDFS,
> >>>>>>>
> >>>>>>> which
> >>>>>>>
> >>>>>>> should
> >>>>>>> be required to run Flink in production, so I guess that would be
> >>>>>>> acceptable for most
> >>>>>>> users. WDYT?
> >>>>>>>
> >>>>>>> WRT testing, demo, and local setups, I think we could support the
> >>>>>>>
> >>>>>>> local
> >>>>>>>
> >>>>>>> filesystem
> >>>>>>> scheme i.e. file://** <file:///**> <file:///**> <> as the state
> backends do. It works as long as
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> Gateway
> >>>>>>> and JobManager(or SQL Driver) can access the resource directory
> >>>>>>>
> >>>>>>> (specified
> >>>>>>>
> >>>>>>> via
> >>>>>>> `sql-gateway.application.storage-dir`).
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> [1]
> >>>>
> >>>>
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.javahttps://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> >>>> <
> >>>>
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.javahttps://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> >>>>
> >>>>>>> [2]
> >>>>
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>> <
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>>
> >>>>>>> [3]
> >>>>
> >>>>
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> >>>> <
> >>>>
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> >>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>> 2023年6月3日 12:21，Jark Wu <imjark@gmail.com <ma...@gmail.com>
> mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>>
> >>>>>>> 写道：
> >>>>>>>
> >>>>>>> Hi Paul,
> >>>>>>>
> >>>>>>> Thanks for your reply. I left my comments inline.
> >>>>>>>
> >>>>>>> As the FLIP said, it’s good to have a default main class for Flink
> >>>>>>>
> >>>>>>> SQLs,
> >>>>>>>
> >>>>>>> which allows users to submit Flink SQLs in the same way as
> >>>>>>>
> >>>>>>> DataStream
> >>>>>>>
> >>>>>>> jobs, or else users need to write their own main class.
> >>>>>>>
> >>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> >>>>>>> DataStream API also doesn't provide a default main class for users,
> >>>>>>> why do we need to provide such one for SQL?
> >>>>>>>
> >>>>>>> With the help of ExecNodeGraph, do we still need the serialized
> >>>>>>> SessionState? If not, we could make SQL Driver accepts two
> >>>>>>>
> >>>>>>> serialized
> >>>>>>>
> >>>>>>> formats:
> >>>>>>>
> >>>>>>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
> >>>>>>>
> >>>>>>> the
> >>>>>>>
> >>>>>>> proposed SqlDriver was a dedicated main class accepting SQL files,
> >>>>>>>
> >>>>>>> is
> >>>>>>>
> >>>>>>> that correct?
> >>>>>>> If true, we have to ship the SessionState for this case which is a
> >>>>>>>
> >>>>>>> large
> >>>>>>>
> >>>>>>> work.
> >>>>>>> I think we just need a JsonPlanDriver which is a main class that
> >>>>>>>
> >>>>>>> accepts
> >>>>>>>
> >>>>>>> JsonPlan as the parameter.
> >>>>>>>
> >>>>>>> The common solutions I know is to use distributed file systems or
> >>>>>>>
> >>>>>>> use
> >>>>>>>
> >>>>>>> init containers to localize the resources.
> >>>>>>>
> >>>>>>> Personally, I prefer the way of init containers which doesn't
> >>>>>>>
> >>>>>>> depend
> >>>>>>>
> >>>>>>> on
> >>>>>>>
> >>>>>>> additional components.
> >>>>>>> This can reduce the moving parts of a production environment.
> >>>>>>> Depending on a distributed file system makes the testing, demo, and
> >>>>>>>
> >>>>>>> local
> >>>>>>>
> >>>>>>> setup harder than init containers.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Jark
> >>>>>>>
> >>>>>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com>
> >>>>>>>
> >>>>>>> <mailto:
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>> <mailto:
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com> <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>>> wrote:
> >>>>>>>
> >>>>>>> The FLIP is in the early phase and some details are not included,
> >>>>>>>
> >>>>>>> but
> >>>>>>>
> >>>>>>> fortunately, we got lots of valuable ideas from the discussion.
> >>>>>>>
> >>>>>>> Thanks to everyone who joined the dissuasion!
> >>>>>>> @Weihua @Shanmon @Shengkai @Biao @Jark
> >>>>>>>
> >>>>>>> This weekend I’m gonna revisit and update the FLIP, adding more
> >>>>>>> details. Hopefully, we can further align our opinions.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com> mailto:paullin3280@gmail.com <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com> <mailto:
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>> 写道：
> >>>>>>>
> >>>>>>> Hi Jark,
> >>>>>>>
> >>>>>>> Thanks a lot for your input!
> >>>>>>>
> >>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> >>>>>>>
> >>>>>>> still
> >>>>>>>
> >>>>>>> necessary to support SQL Driver?
> >>>>>>>
> >>>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
> >>>>>>>
> >>>>>>> simplify
> >>>>>>>
> >>>>>>> Flink SQL execution with Flink CLI.
> >>>>>>>
> >>>>>>> As the FLIP said, it’s good to have a default main class for
> >>>>>>>
> >>>>>>> Flink
> >>>>>>>
> >>>>>>> SQLs,
> >>>>>>>
> >>>>>>> which allows users to submit Flink SQLs in the same way as
> >>>>>>>
> >>>>>>> DataStream
> >>>>>>>
> >>>>>>> jobs, or else users need to write their own main class.
> >>>>>>>
> >>>>>>> SQL Driver needs to serialize SessionState which is very
> >>>>>>>
> >>>>>>> challenging
> >>>>>>>
> >>>>>>> but not detailed covered in the FLIP.
> >>>>>>>
> >>>>>>> With the help of ExecNodeGraph, do we still need the serialized
> >>>>>>> SessionState? If not, we could make SQL Driver accepts two
> >>>>>>>
> >>>>>>> serialized
> >>>>>>>
> >>>>>>> formats:
> >>>>>>>
> >>>>>>> - SQL files for user-facing public usage
> >>>>>>> - ExecNodeGraph for internal usage
> >>>>>>>
> >>>>>>> It’s kind of similar to the relationship between job jars and
> >>>>>>>
> >>>>>>> jobgraphs.
> >>>>>>>
> >>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> >>>>>>>
> >>>>>>> true?
> >>>>>>>
> >>>>>>> Is it
> >>>>>>>
> >>>>>>> possible to support it?
> >>>>>>>
> >>>>>>> Yes, K8s doesn’t distribute any files. It’s the users’
> >>>>>>>
> >>>>>>> responsibility
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> make
> >>>>>>>
> >>>>>>> sure the resources are accessible in the containers. The common
> >>>>>>>
> >>>>>>> solutions
> >>>>>>>
> >>>>>>> I know is to use distributed file systems or use init containers
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> localize the
> >>>>>>>
> >>>>>>> resources.
> >>>>>>>
> >>>>>>> Now I lean toward introducing a fs to do the distribution job.
> >>>>>>>
> >>>>>>> WDYT?
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com>
> mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>
> >>>>>>> <mailto:
> >>>>>>>
> >>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com
> <mailto:imjark@gmail.com mailto:imjark@gmail.com>>
> >>>>>>>
> >>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com <mailto:
> imjark@gmail.com mailto:imjark@gmail.com> <mailto:
> >>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com
> <mailto:imjark@gmail.com mailto:imjark@gmail.com>>>
> >>>>>>>
> >>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com <mailto:
> imjark@gmail.com mailto:imjark@gmail.com> <mailto:
> >>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com
> <mailto:imjark@gmail.com mailto:imjark@gmail.com>> <mailto:
> >>>>>>>
> >>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com
> <mailto:imjark@gmail.com mailto:imjark@gmail.com> <mailto:
> imjark@gmail.commailto:imjark@gmail.com
> >>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com>>>>>
> >>>>>>>
> >>>>>>> 写道：
> >>>>>>>
> >>>>>>> Hi Paul,
> >>>>>>>
> >>>>>>> Thanks for starting this discussion. I like the proposal! This
> >>>>>>>
> >>>>>>> is
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>> frequently requested feature!
> >>>>>>>
> >>>>>>> I agree with Shengkai that ExecNodeGraph as the submission
> >>>>>>>
> >>>>>>> object
> >>>>>>>
> >>>>>>> is a
> >>>>>>>
> >>>>>>> better idea than SQL file. To be more specific, it should be
> >>>>>>>
> >>>>>>> JsonPlanGraph
> >>>>>>>
> >>>>>>> or CompiledPlan which is the serializable representation.
> >>>>>>>
> >>>>>>> CompiledPlan
> >>>>>>>
> >>>>>>> is a
> >>>>>>>
> >>>>>>> clear separation between compiling/optimization/validation and
> >>>>>>>
> >>>>>>> execution.
> >>>>>>>
> >>>>>>> This can keep the validation and metadata accessing still on the
> >>>>>>>
> >>>>>>> SQLGateway
> >>>>>>>
> >>>>>>> side. This allows SQLGateway to leverage some metadata caching
> >>>>>>>
> >>>>>>> and
> >>>>>>>
> >>>>>>> UDF
> >>>>>>>
> >>>>>>> JAR
> >>>>>>>
> >>>>>>> caching for better compiling performance.
> >>>>>>>
> >>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> >>>>>>>
> >>>>>>> still
> >>>>>>>
> >>>>>>> necessary to support SQL Driver? Regarding non-interactive SQL
> >>>>>>>
> >>>>>>> jobs,
> >>>>>>>
> >>>>>>> users
> >>>>>>>
> >>>>>>> can use the Table API program for application mode. SQL Driver
> >>>>>>>
> >>>>>>> needs
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> serialize SessionState which is very challenging but not
> >>>>>>>
> >>>>>>> detailed
> >>>>>>>
> >>>>>>> covered
> >>>>>>>
> >>>>>>> in the FLIP.
> >>>>>>>
> >>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> >>>>>>>
> >>>>>>> true?
> >>>>>>>
> >>>>>>> Is it
> >>>>>>>
> >>>>>>> possible to support it?
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Jark
> >>>>>>>
> >>>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com>
> >>>>>>>
> >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> >>>>>>> <mailto:
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com> <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>> <mailto:
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com> <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>> <mailto:
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com> <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi Weihua,
> >>>>>>>
> >>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
> >>>>>>>
> >>>>>>> challenging
> >>>>>>>
> >>>>>>> parts of this FLIP.
> >>>>>>>
> >>>>>>> Web submission is not enabled in application mode currently as
> >>>>>>>
> >>>>>>> you
> >>>>>>>
> >>>>>>> said,
> >>>>>>>
> >>>>>>> but it could be changed if we have good reasons.
> >>>>>>>
> >>>>>>> What do you think about introducing a distributed storage for
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> Gateway?
> >>>>>>>
> >>>>>>> We could make use of Flink file systems [1] to distribute the
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> Gateway
> >>>>>>>
> >>>>>>> generated resources, that should solve the problem at its root
> >>>>>>>
> >>>>>>> cause.
> >>>>>>>
> >>>>>>> Users could specify Flink-supported file systems to ship files.
> >>>>>>>
> >>>>>>> It’s
> >>>>>>>
> >>>>>>> only
> >>>>>>>
> >>>>>>> required when using SQL Gateway with K8s application mode.
> >>>>>>>
> >>>>>>> [1]
> >>>>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>> <
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>> <
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>> <
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>> <
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>> <
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>> <
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>> <
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>> <
> >>>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> huweihua.ckl@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:
> >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> huweihua.ckl@gmail.com> <mailto:
> >>>>>>>
> >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> huweihua.ckl@gmail.com>> <mailto:
> >>>>>>>
> >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> huweihua.ckl@gmail.com> <mailto:
> >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> huweihua.ckl@gmail.com>>>> 写道：
> >>>>>>>
> >>>>>>> Thanks Paul for your reply.
> >>>>>>>
> >>>>>>> SQLDriver looks good to me.
> >>>>>>>
> >>>>>>> 2. Do you mean a pass the SQL string a configuration or a
> >>>>>>>
> >>>>>>> program
> >>>>>>>
> >>>>>>> argument?
> >>>>>>>
> >>>>>>> I brought this up because we were unable to pass the SQL file
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> Flink
> >>>>>>>
> >>>>>>> using Kubernetes mode.
> >>>>>>> For DataStream/Python users, they need to prepare their images
> >>>>>>>
> >>>>>>> for
> >>>>>>>
> >>>>>>> the
> >>>>>>>
> >>>>>>> jars
> >>>>>>>
> >>>>>>> and dependencies.
> >>>>>>> But for SQL users, they can use a common image to run
> >>>>>>>
> >>>>>>> different
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> queries
> >>>>>>>
> >>>>>>> if there are no other udf requirements.
> >>>>>>> It would be great if the SQL query and image were not bound.
> >>>>>>>
> >>>>>>> Using strings is a way to decouple these, but just as you
> >>>>>>>
> >>>>>>> mentioned,
> >>>>>>>
> >>>>>>> it's
> >>>>>>>
> >>>>>>> not easy to pass complex SQL.
> >>>>>>>
> >>>>>>> use web submission
> >>>>>>>
> >>>>>>> AFAIK, we can not use web submission in the Application mode.
> >>>>>>>
> >>>>>>> Please
> >>>>>>>
> >>>>>>> correct me if I'm wrong.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Weihua
> >>>>>>>
> >>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>
> >>>>>>>
> >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> >>>>>>>
> >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>>>
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi Biao,
> >>>>>>>
> >>>>>>> Thanks for your comments!
> >>>>>>>
> >>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
> >>>>>>>
> >>>>>>> Flink
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> jobs
> >>>>>>>
> >>>>>>> in
> >>>>>>>
> >>>>>>> Application mode? More specifically, if we use SQL
> >>>>>>>
> >>>>>>> client/gateway
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> execute some interactive SQLs like a SELECT query, can we
> >>>>>>>
> >>>>>>> ask
> >>>>>>>
> >>>>>>> flink
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> use
> >>>>>>>
> >>>>>>> Application mode to execute those queries after this FLIP?
> >>>>>>>
> >>>>>>> Thanks for pointing it out. I think only DMLs would be
> >>>>>>>
> >>>>>>> executed
> >>>>>>>
> >>>>>>> via
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> Driver.
> >>>>>>> I'll add the scope to the FLIP.
> >>>>>>>
> >>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> >>>>>>>
> >>>>>>> trivial as
> >>>>>>>
> >>>>>>> we
> >>>>>>>
> >>>>>>> can ship files via YARN's tool easily but for K8s, things
> >>>>>>>
> >>>>>>> can
> >>>>>>>
> >>>>>>> be
> >>>>>>>
> >>>>>>> more
> >>>>>>>
> >>>>>>> complicated as Shengkai said.
> >>>>>>>
> >>>>>>> Your input is very informative. I’m thinking about using web
> >>>>>>>
> >>>>>>> submission,
> >>>>>>>
> >>>>>>> but it requires exposing the JobManager port which could also
> >>>>>>>
> >>>>>>> be
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>> problem
> >>>>>>>
> >>>>>>> on K8s.
> >>>>>>>
> >>>>>>> Another approach is to explicitly require a distributed
> >>>>>>>
> >>>>>>> storage
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> ship
> >>>>>>>
> >>>>>>> files,
> >>>>>>> but we may need a new deployment executor for that.
> >>>>>>>
> >>>>>>> What do you think of these two approaches?
> >>>>>>>
> >>>>>>> 3. Serialization of SessionState: in SessionState, there are
> >>>>>>>
> >>>>>>> some
> >>>>>>>
> >>>>>>> unserializable fields
> >>>>>>> like
> >>>>>>>
> >>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >>>>>>>
> >>>>>>> It
> >>>>>>>
> >>>>>>> may be worthwhile to add more details about the
> >>>>>>>
> >>>>>>> serialization
> >>>>>>>
> >>>>>>> part.
> >>>>>>>
> >>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph
> >>>>>>>
> >>>>>>> as
> >>>>>>>
> >>>>>>> Shengkai
> >>>>>>>
> >>>>>>> mentioned, do we eliminate the need for serialization of
> >>>>>>>
> >>>>>>> SessionState?
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
> biaogeng7@gmail.com> mailto:biaogeng7@gmail.com <mailto:
> >>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:
> biaogeng7@gmail.com> <mailto:
> >>>>>>>
> >>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:
> biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com>>
> <mailto:
> >>>>>>>
> >>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:
> biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com>
> <mailto:
> >>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:
> biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com>>>>
> 写道：
> >>>>>>>
> >>>>>>> Thanks Paul for the proposal!I believe it would be very
> >>>>>>>
> >>>>>>> useful
> >>>>>>>
> >>>>>>> for
> >>>>>>>
> >>>>>>> flink
> >>>>>>>
> >>>>>>> users.
> >>>>>>> After reading the FLIP, I have some questions:
> >>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
> >>>>>>>
> >>>>>>> Flink
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> jobs
> >>>>>>>
> >>>>>>> in
> >>>>>>>
> >>>>>>> Application mode? More specifically, if we use SQL
> >>>>>>>
> >>>>>>> client/gateway
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> execute some interactive SQLs like a SELECT query, can we
> >>>>>>>
> >>>>>>> ask
> >>>>>>>
> >>>>>>> flink
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> use
> >>>>>>>
> >>>>>>> Application mode to execute those queries after this FLIP?
> >>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> >>>>>>>
> >>>>>>> trivial as
> >>>>>>>
> >>>>>>> we
> >>>>>>>
> >>>>>>> can ship files via YARN's tool easily but for K8s, things
> >>>>>>>
> >>>>>>> can
> >>>>>>>
> >>>>>>> be
> >>>>>>>
> >>>>>>> more
> >>>>>>>
> >>>>>>> complicated as Shengkai said. I have implemented a simple
> >>>>>>>
> >>>>>>> POC
> >>>>>>>
> >>>>>>> <
> >>>>
> >>>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>> <
> >>>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>> <
> >>>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>> <
> >>>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>> <
> >>>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>>
> >>>>>>> based on SQL client before(i.e. consider the SQL client
> >>>>>>>
> >>>>>>> which
> >>>>>>>
> >>>>>>> supports
> >>>>>>>
> >>>>>>> executing a SQL file as the SQL driver in this FLIP). One
> >>>>>>>
> >>>>>>> problem
> >>>>>>>
> >>>>>>> I
> >>>>>>>
> >>>>>>> have
> >>>>>>>
> >>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s
> >>>>>>>
> >>>>>>> side.
> >>>>>>>
> >>>>>>> Without
> >>>>>>>
> >>>>>>> such support, users have to modify the initContainer or
> >>>>>>>
> >>>>>>> rebuild
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>> new
> >>>>>>>
> >>>>>>> K8s
> >>>>>>>
> >>>>>>> image every time to fetch the SQL file. Like the flink k8s
> >>>>>>>
> >>>>>>> operator,
> >>>>>>>
> >>>>>>> one
> >>>>>>>
> >>>>>>> workaround is to utilize the flink config(transforming the
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> file
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>> escaped string like Weihua mentioned) which will be
> >>>>>>>
> >>>>>>> converted
> >>>>>>>
> >>>>>>> to a
> >>>>>>>
> >>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger
> >>>>>>>
> >>>>>>> than
> >>>>>>>
> >>>>>>> 1MB
> >>>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/
> https://kubernetes.io/docs/concepts/configuration/configmap/ <
> >>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/
> https://kubernetes.io/docs/concepts/configuration/configmap/>
> >>>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/
> https://kubernetes.io/docs/concepts/configuration/configmap/ <
> >>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/
> https://kubernetes.io/docs/concepts/configuration/configmap/>> <
> >>>>>>>
> >>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/
> https://kubernetes.io/docs/concepts/configuration/configmap/ <
> >>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/
> https://kubernetes.io/docs/concepts/configuration/configmap/> <
> >>>>>>>
> >>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/
> https://kubernetes.io/docs/concepts/configuration/configmap/ <
> >>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/
> https://kubernetes.io/docs/concepts/configuration/configmap/>>>>).
> >>>>>>>
> >>>>>>> Not
> >>>>>>>
> >>>>>>> sure
> >>>>>>>
> >>>>>>> if we have better solutions.
> >>>>>>> 3. Serialization of SessionState: in SessionState, there are
> >>>>>>>
> >>>>>>> some
> >>>>>>>
> >>>>>>> unserializable fields
> >>>>>>> like
> >>>>>>>
> >>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >>>>>>>
> >>>>>>> It
> >>>>>>>
> >>>>>>> may be worthwhile to add more details about the
> >>>>>>>
> >>>>>>> serialization
> >>>>>>>
> >>>>>>> part.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Biao Geng
> >>>>>>>
> >>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>
> >>>>>>> <mailto:
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>
> >>>>>>>
> >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>>>
> >>>>>>>
> >>>>>>> 于2023年5月31日周三 11:49写道：
> >>>>>>>
> >>>>>>> Hi Weihua,
> >>>>>>>
> >>>>>>> Thanks a lot for your input! Please see my comments inline.
> >>>>>>>
> >>>>>>> - Is SQLRunner the better name? We use this to run a SQL
> >>>>>>>
> >>>>>>> Job.
> >>>>>>>
> >>>>>>> (Not
> >>>>>>>
> >>>>>>> strong,
> >>>>>>>
> >>>>>>> the SQLDriver is fine for me)
> >>>>>>>
> >>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
> >>>>>>>
> >>>>>>> following
> >>>>>>>
> >>>>>>> reasons FYI:
> >>>>>>>
> >>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
> >>>>>>> 2. Flink program's main class is sort of like Driver in
> >>>>>>>
> >>>>>>> JDBC
> >>>>>>>
> >>>>>>> which
> >>>>>>>
> >>>>>>> translates SQLs into
> >>>>>>> databases specific languages.
> >>>>>>>
> >>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >>>>>>>
> >>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
> >>>>>>>
> >>>>>>> we
> >>>>>>>
> >>>>>>> need
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> prepare
> >>>>>>>
> >>>>>>> a SQL file in an image for Kubernetes application mode,
> >>>>>>>
> >>>>>>> which
> >>>>>>>
> >>>>>>> may
> >>>>>>>
> >>>>>>> be
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>> bit
> >>>>>>>
> >>>>>>> cumbersome.
> >>>>>>>
> >>>>>>> Do you mean a pass the SQL string a configuration or a
> >>>>>>>
> >>>>>>> program
> >>>>>>>
> >>>>>>> argument?
> >>>>>>>
> >>>>>>> I thought it might be convenient for testing propose, but
> >>>>>>>
> >>>>>>> not
> >>>>>>>
> >>>>>>> recommended
> >>>>>>>
> >>>>>>> for production,
> >>>>>>> cause Flink SQLs could be complicated and involves lots of
> >>>>>>>
> >>>>>>> characters
> >>>>>>>
> >>>>>>> that
> >>>>>>>
> >>>>>>> need to escape.
> >>>>>>>
> >>>>>>> WDYT?
> >>>>>>>
> >>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>>>>>
> >>>>>>> "run-application"
> >>>>>>>
> >>>>>>> command. Does that mean we need to perform automatic
> >>>>>>>
> >>>>>>> detection
> >>>>>>>
> >>>>>>> in
> >>>>>>>
> >>>>>>> Flink?
> >>>>>>>
> >>>>>>> Yes! It’s like running a PyFlink job with the following
> >>>>>>>
> >>>>>>> command:
> >>>>>>>
> >>>>>>> `./bin/flink run \\ --pyModule table.word_count \\ --pyFiles
> examples/python/table`
> >>>>>>>
> >>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
> >>>>>>>
> >>>>>>> Driver
> >>>>>>>
> >>>>>>> automatically.
> >>>>>>>
> >>>>>>> [1]
> >>>>
> >>>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>> <
> >>>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>> <
> >>>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>> <
> >>>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>> <
> >>>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> huweihua.ckl@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:
> >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> huweihua.ckl@gmail.com>
> >>>>>>>
> >>>>>>> <mailto:
> >>>>>>>
> >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> huweihua.ckl@gmail.com>> <mailto:
> >>>>>>>
> >>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:
> huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:
> huweihua.ckl@gmail.com>>> 写道：
> >>>>>>>
> >>>>>>> Thanks Paul for the proposal.
> >>>>>>>
> >>>>>>> +1 for this. It is valuable in improving ease of use.
> >>>>>>>
> >>>>>>> I have a few questions.
> >>>>>>> - Is SQLRunner the better name? We use this to run a SQL
> >>>>>>>
> >>>>>>> Job.
> >>>>>>>
> >>>>>>> (Not
> >>>>>>>
> >>>>>>> strong,
> >>>>>>>
> >>>>>>> the SQLDriver is fine for me)
> >>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
> >>>>>>>
> >>>>>>> we
> >>>>>>>
> >>>>>>> need
> >>>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>> prepare
> >>>>>>>
> >>>>>>> a SQL file in an image for Kubernetes application mode,
> >>>>>>>
> >>>>>>> which
> >>>>>>>
> >>>>>>> may
> >>>>>>>
> >>>>>>> be
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>> bit
> >>>>>>>
> >>>>>>> cumbersome.
> >>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>>>>>
> >>>>>>> "run-application"
> >>>>>>>
> >>>>>>> command. Does that mean we need to perform automatic
> >>>>>>>
> >>>>>>> detection
> >>>>>>>
> >>>>>>> in
> >>>>>>>
> >>>>>>> Flink?
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Weihua
> >>>>>>>
> >>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> >>>>>>>
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>
> >>>>>>>
> >>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com
> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> >>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:
> paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:
> paullin3280@gmail.com>>>>
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi team,
> >>>>>>>
> >>>>>>> I’d like to start a discussion about FLIP-316 [1], which
> >>>>>>>
> >>>>>>> introduces
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> driver as the
> >>>>>>> default main class for Flink SQL jobs.
> >>>>>>>
> >>>>>>> Currently, Flink SQL could be executed out of the box
> >>>>>>>
> >>>>>>> either
> >>>>>>>
> >>>>>>> via
> >>>>>>>
> >>>>>>> SQL
> >>>>>>>
> >>>>>>> Client/Gateway
> >>>>>>> or embedded in a Flink Java/Python program.
> >>>>>>>
> >>>>>>> However, each one has its drawback:
> >>>>>>>
> >>>>>>> - SQL Client/Gateway doesn’t support the application
> >>>>>>>
> >>>>>>> deployment
> >>>>>>>
> >>>>>>> mode
> >>>>>>>
> >>>>>>> [2]
> >>>>>>>
> >>>>>>> - Flink Java/Python program requires extra work to write
> >>>>>>>
> >>>>>>> a
> >>>>>>>
> >>>>>>> non-SQL
> >>>>>>>
> >>>>>>> program
> >>>>>>>
> >>>>>>> Therefore, I propose adding a SQL driver to act as the
> >>>>>>>
> >>>>>>> default
> >>>>>>>
> >>>>>>> main
> >>>>>>>
> >>>>>>> class
> >>>>>>>
> >>>>>>> for SQL jobs.
> >>>>>>> Please see the FLIP docs for details and feel free to
> >>>>>>>
> >>>>>>> comment.
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> [1]
> >>>>
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >>>> <
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >>>>
> >>>>>>> <
> >>>>
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>> <
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>>
> >>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541
> https://issues.apache.org/jira/browse/FLINK-26541 <
> >>>>>>> https://issues.apache.org/jira/browse/FLINK-26541
> https://issues.apache.org/jira/browse/FLINK-26541> <
> >>>>>>> https://issues.apache.org/jira/browse/FLINK-26541
> https://issues.apache.org/jira/browse/FLINK-26541 <
> >>>>>>> https://issues.apache.org/jira/browse/FLINK-26541
> https://issues.apache.org/jira/browse/FLINK-26541>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Ferenc,
Sorry for my late reply.
> Is any active work happening on this FLIP? As far as I see there
> are blockers that needs to happen first to implement regarding
> artifact distribution.
You’re right. There’s a block in K8s application mode, but none in
YARN application. I’m doing a POC on YARN application mode
before starting a vote thread.
I’ve been busy lately, but the FLIP is active for sure. The situation
will change in a couple of weeks.
Thank you for reaching out! I’ll let you know when the POC is completed.
Best,
Paul Lam
> 2023年11月21日 06:01，Ferenc Csaky <fe...@pm.me.INVALID> 写道：
>
> Hello devs,
>
> Is any active work happening on this FLIP? As far as I see there
> are blockers that needs to happen first to implement regarding
> artifact distribution.
>
> Is this work in halt completetly or some efforts are going into
> resolve the blockers first or something?
>
> Our platform would benefit this feature a lot, we have a kind of
> working custom implementation at the moment, but it is uniquely
> adapted to our app and platform.
>
> I could help out to move this forward.
>
> Best,
> Ferenc
>
>
>
> On Friday, June 30th, 2023 at 04:53, Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
>
>
>>
>>
>> Hi Jing,
>>
>> Thanks for your input!
>>
>>> Would you like to add
>>> one section to describe(better with script/code example) how to use it in
>>> these two scenarios from users' perspective?
>>
>>
>> OK. I’ll update the FLIP with the code snippet after I get the POC branch done.
>>
>>> NIT: the pictures have transparent background when readers click on it. It
>>> would be great if you can replace them with pictures with white background.
>>
>>
>> Fixed. Thanks for pointing that out :)
>>
>> Best,
>> Paul Lam
>>
>>> 2023年6月27日 06:51，Jing Ge jing@ververica.com.INVALID 写道：
>>>
>>> Hi Paul,
>>>
>>> Thanks for driving it and thank you all for the informative discussion! The
>>> FLIP is in good shape now. As described in the FLIP, SQL Driver will be
>>> mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway in
>>> application mode and 2. external system integration. Would you like to add
>>> one section to describe(better with script/code example) how to use it in
>>> these two scenarios from users' perspective?
>>>
>>> NIT: the pictures have transparent background when readers click on it. It
>>> would be great if you can replace them with pictures with white background.
>>>
>>> Best regards,
>>> Jing
>>>
>>> On Mon, Jun 26, 2023 at 1:31 PM Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com> wrote:
>>>
>>>> Hi Shengkai,
>>>>
>>>>> * How can we ship the json plan to the JobManager?
>>>>
>>>> The Flink K8s module should be responsible for file distribution. We could
>>>> introduce
>>>> an option like `kubernetes.storage.dir`. For each flink cluster, there
>>>> would be a
>>>> dedicated subdirectory, with the pattern like
>>>> `${kubernetes.storage.dir}/${cluster-id}`.
>>>>
>>>> All resources-related options (e.g. pipeline jars, json plans) that are
>>>> configured with
>>>> scheme `file://` <file://`/> <file://`/> <file:///%60 <file:///%60>> would be uploaded to the resource directory
>>>> and downloaded to the
>>>> jobmanager, before SQL Driver accesses the files with the original
>>>> filenames.
>>>>
>>>>> * Classloading strategy
>>>>
>>>> We could directly specify the SQL Gateway jar as the jar file in
>>>> PackagedProgram.
>>>> It would be treated like a normal user jar and the SQL Driver is loaded
>>>> into the user
>>>> classloader. WDYT?
>>>>
>>>>> * Option `$internal.sql-gateway.driver.sql-config` is string type
>>>>> I think it's better to use Map type here
>>>>
>>>> By Map type configuration, do you mean a nested map that contains all
>>>> configurations?
>>>>
>>>> I hope I've explained myself well, it’s a file that contains the extra SQL
>>>> configurations, which would be shipped to the jobmanager.
>>>>
>>>>> * PoC branch
>>>>
>>>> Sure. I’ll let you know once I get the job done.
>>>>
>>>> Best,
>>>> Paul Lam
>>>>
>>>>> 2023年6月26日 14:27，Shengkai Fang <fskmine@gmail.com <ma...@gmail.com> mailto:fskmine@gmail.com> 写道：
>>>>>
>>>>> Hi, Paul.
>>>>>
>>>>> Thanks for your update. I have a few questions about the new design:
>>>>>
>>>>> * How can we ship the json plan to the JobManager?
>>>>>
>>>>> The current design only exposes an option about the URL of the json
>>>>> plan. It seems the gateway is responsible to upload to an external stroage.
>>>>> Can we reuse the PipelineOptions.JARS to ship to the remote filesystem?
>>>>>
>>>>> * Classloading strategy
>>>>>
>>>>> Currently, the Driver is in the sql-gateway package. It means the Driver
>>>>> is not in the JM's classpath directly. Because the sql-gateway jar is now
>>>>> in the opt directory rather than lib directory. It may need to add the
>>>>> external dependencies as Python does[1]. BTW, I think it's better to move
>>>>> the Driver into the flink-table-runtime package, which is much easier to
>>>>> find(Sorry for the wrong opinion before).
>>>>>
>>>>> * Option `$internal.sql-gateway.driver.sql-config` is string type
>>>>>
>>>>> I think it's better to use Map type here
>>>>>
>>>>> * PoC branch
>>>>>
>>>>> Because this FLIP involves many modules, do you have a PoC branch to
>>>>> verify it does work?
>>>>>
>>>>> Best,
>>>>> Shengkai
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
>>>>> <
>>>>> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
>>>>>
>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
>>>>> 于2023年6月19日周一 14:09写道：
>>>>> Hi Shengkai,
>>>>>
>>>>> Sorry for my late reply. It took me some time to update the FLIP.
>>>>>
>>>>> In the latest FLIP design, SQL Driver is placed in flink-sql-gateway
>>>>> module. PTAL.
>>>>>
>>>>> The FLIP does not cover details about the K8s file distribution, but its
>>>>> general usage would
>>>>> be very much the same as YARN setups. We could make follow-up
>>>>> discussions in the jira
>>>>> tickets.
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>>> 2023年6月12日 15:29，Shengkai Fang <fskmine@gmail.com <ma...@gmail.com> mailto:fskmine@gmail.com <mailto:
>>>>>> fskmine@gmail.com <ma...@gmail.com> mailto:fskmine@gmail.com>> 写道：
>>>>>>
>>>>>>> If it’s the case, I’m good with introducing a new module and making
>>>>>>> SQL Driver
>>>>>>> an internal class and accepts JSON plans only.
>>>>>>
>>>>>> I rethink this again and again. I think it's better to move the
>>>>>> SqlDriver into the sql-gateway module because the sql client relies on the
>>>>>> sql-gateway to submit the sql and the sql-gateway has the ability to
>>>>>> generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
>>>>>>
>>>>>> * Upload configuration through command line parameter
>>>>>>
>>>>>> ExecNodeGraph only contains the job's information but it doesn't
>>>>>> contain the checkpoint dir, checkpoint interval, execution mode and so on.
>>>>>> So I think we should also upload the configuration.
>>>>>>
>>>>>> * KubernetesClusterDescripter and
>>>>>> KubernetesApplicationClusterEntrypoint are responsible for the jar
>>>>>> upload/download
>>>>>>
>>>>>> +1 for the change.
>>>>>>
>>>>>> Could you update the FLIP about the current discussion?
>>>>>>
>>>>>> Best,
>>>>>> Shengkai
>>>>>>
>>>>>> Yang Wang <wangyang0918@apache.org <ma...@apache.org> mailto:wangyang0918@apache.org <ma...@apache.org>>
>>>>>> 于2023年6月12日周一 11:41写道：
>>>>>> Sorry for the late reply. I am in favor of introducing such a built-in
>>>>>> resource localization mechanism
>>>>>> based on Flink FileSystem. Then FLINK-28915[1] could be the second step
>>>>>> which will download
>>>>>> the jars and dependencies to the JobManager/TaskManager local directory
>>>>>> before working.
>>>>>>
>>>>>> The first step could be done in another ticket in Flink. Or some
>>>>>> external
>>>>>> Flink jobs management system
>>>>>> could also take care of this.
>>>>>>
>>>>>> [1]. https://issues.apache.org/jira/browse/FLINK-28915 https://issues.apache.org/jira/browse/FLINK-28915 <
>>>>>> https://issues.apache.org/jira/browse/FLINK-28915 https://issues.apache.org/jira/browse/FLINK-28915>
>>>>>>
>>>>>> Best,
>>>>>> Yang
>>>>>>
>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
>>>>>> 于2023年6月9日周五 17:39写道：
>>>>>>
>>>>>>> Hi Mason,
>>>>>>>
>>>>>>> I get your point. I'm increasingly feeling the need to introduce a
>>>>>>> built-in
>>>>>>> file distribution mechanism for flink-kubernetes module, just like
>>>>>>> Spark
>>>>>>> does with `spark.kubernetes.file.upload.path` [1].
>>>>>>>
>>>>>>> I’m assuming the workflow is as follows:
>>>>>>>
>>>>>>> - KubernetesClusterDescripter uploads all local resources to a remote
>>>>>>> storage via Flink filesystem (skips if the resources are already
>>>>>>> remote).
>>>>>>> - KubernetesApplicationClusterEntrypoint downloads the resources
>>>>>>> and put them in the classpath during startup.
>>>>>>>
>>>>>>> I wouldn't mind splitting it into another FLIP to ensure that
>>>>>>> everything is
>>>>>>> done correctly.
>>>>>>>
>>>>>>> cc'ed @Yang to gather more opinions.
>>>>>>>
>>>>>>> [1]
>>>>
>>>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
>>>> <
>>>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>> 2023年6月8日 12:15，Mason Chen <mas.chen6345@gmail.com <ma...@gmail.com> mailto:mas.chen6345@gmail.com <mailto:
>>>>>>> mas.chen6345@gmail.com <ma...@gmail.com> mailto:mas.chen6345@gmail.com>> 写道：
>>>>>>>
>>>>>>> Hi Paul,
>>>>>>>
>>>>>>> Thanks for your response!
>>>>>>>
>>>>>>> I agree that utilizing SQL Drivers in Java applications is equally
>>>>>>> important
>>>>>>>
>>>>>>> as employing them in SQL Gateway. WRT init containers, I think most
>>>>>>> users use them just as a workaround. For example, wget a jar from the
>>>>>>> maven repo.
>>>>>>>
>>>>>>> We could implement the functionality in SQL Driver in a more graceful
>>>>>>> way and the flink-supported filesystem approach seems to be a
>>>>>>> good choice.
>>>>>>>
>>>>>>> My main point is: can we solve the problem with a design agnostic of
>>>>>>> SQL
>>>>>>> and Stream API? I mentioned a use case where this ability is useful
>>>>>>> for
>>>>>>> Java or Stream API applications. Maybe this is even a non-goal to
>>>>>>> your FLIP
>>>>>>> since you are focusing on the driver entrypoint.
>>>>>>>
>>>>>>> Jark mentioned some optimizations:
>>>>>>>
>>>>>>> This allows SQLGateway to leverage some metadata caching and UDF JAR
>>>>>>> caching for better compiling performance.
>>>>>>>
>>>>>>> It would be great to see this even outside the SQLGateway (i.e. UDF
>>>>>>> JAR
>>>>>>> caching).
>>>>>>>
>>>>>>> Best,
>>>>>>> Mason
>>>>>>>
>>>>>>> On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskmine@gmail.com <ma...@gmail.com> mailto:fskmine@gmail.com
>>>>>>> <mailto:fskmine@gmail.com mailto:fskmine@gmail.com>> wrote:
>>>>>>>
>>>>>>> Hi. Paul. Thanks for your update and the update makes me understand
>>>>>>> the
>>>>>>> design much better.
>>>>>>>
>>>>>>> But I still have some questions about the FLIP.
>>>>>>>
>>>>>>> For SQL Gateway, only DMLs need to be delegated to the SQL server
>>>>>>> Driver. I would think about the details and update the FLIP. Do you
>>>>>>> have
>>>>>>>
>>>>>>> some
>>>>>>>
>>>>>>> ideas already?
>>>>>>>
>>>>>>> If the applicaiton mode can not support library mode, I think we
>>>>>>> should
>>>>>>> only execute INSERT INTO and UPDATE/ DELETE statement in the
>>>>>>> application
>>>>>>> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
>>>>>>> statements. The ANALYZE TABLE syntax need to register the statistic
>>>>>>> to the
>>>>>>> catalog after job finishes and the CALL PROCEDURE statement doesn't
>>>>>>> generate the ExecNodeGraph.
>>>>>>>
>>>>>>> * Introduce storage via option `sql-gateway.application.storage-dir`
>>>>>>>
>>>>>>> If we can not support to submit the jars through web submission, +1 to
>>>>>>> introduce the options to upload the files. While I think the uploader
>>>>>>> should be responsible to remove the uploaded jars. Can we remove the
>>>>>>> jars
>>>>>>> if the job is running or gateway exits?
>>>>>>>
>>>>>>> * JobID is not avaliable
>>>>>>>
>>>>>>> Can we use the returned rest client by ApplicationDeployer to query
>>>>>>> the job
>>>>>>> id? I am concerned that users don't know which job is related to the
>>>>>>> submitted SQL.
>>>>>>>
>>>>>>> * Do we need to introduce a new module named flink-table-sql-runner?
>>>>>>>
>>>>>>> It seems we need to introduce a new module. Will the new module is
>>>>>>> available in the distribution package? I agree with Jark that we
>>>>>>> don't need
>>>>>>> to introduce this for table-API users and these users have their main
>>>>>>> class. If we want to make users write the k8s operator more easily, I
>>>>>>> think
>>>>>>> we should modify the k8s operator repo. If we don't need to support
>>>>>>> SQL
>>>>>>> files, can we make this jar only visible in the sql-gateway like we
>>>>>>> do in
>>>>>>> the planner loader?[1]
>>>>>>>
>>>>>>> [1]
>>>>
>>>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
>>>> <
>>>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
>>>>
>>>>>>> Best,
>>>>>>> Shengkai
>>>>>>>
>>>>>>> Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>>
>>>>>>> 于2023年6月7日周三 10:52写道：
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks for updating the FLIP.
>>>>>>>
>>>>>>> I have two cents on the distribution of SQLs and resources.
>>>>>>> 1. Should we support a common file distribution mechanism for k8s
>>>>>>> application mode?
>>>>>>> I have seen some issues and requirements on the mailing list.
>>>>>>> In our production environment, we implement the download command in
>>>>>>> the
>>>>>>> CliFrontend.
>>>>>>> And automatically add an init container to the POD for file
>>>>>>>
>>>>>>> downloading.
>>>>>>>
>>>>>>> The advantage of this
>>>>>>> is that we can use all Flink-supported file systems to store files.
>>>>>>>
>>>>>>> This need more discussion. I would appreciate hearing more opinions.
>>>>>>>
>>>>>>> 2. In this FLIP, we distribute files in two different ways in YARN and
>>>>>>> Kubernetes. Can we combine it in one way?
>>>>>>> If we don't want to implement a common file distribution for k8s
>>>>>>> application mode. Could we use the SQLDriver
>>>>>>> to download the files both in YARN and K8S? IMO, this can reduce the
>>>>>>>
>>>>>>> cost
>>>>>>>
>>>>>>> of code maintenance.
>>>>>>>
>>>>>>> Best,
>>>>>>> Weihua
>>>>>>>
>>>>>>> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com
>>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>> wrote:
>>>>>>>
>>>>>>> Hi Mason,
>>>>>>>
>>>>>>> Thanks for your input!
>>>>>>>
>>>>>>> +1 for init containers or a more generalized way of obtaining
>>>>>>>
>>>>>>> arbitrary
>>>>>>>
>>>>>>> files. File fetching isn't specific to just SQL--it also matters for
>>>>>>>
>>>>>>> Java
>>>>>>>
>>>>>>> applications if the user doesn't want to rebuild a Flink image and
>>>>>>>
>>>>>>> just
>>>>>>>
>>>>>>> wants to modify the user application fat jar.
>>>>>>>
>>>>>>> I agree that utilizing SQL Drivers in Java applications is equally
>>>>>>> important
>>>>>>> as employing them in SQL Gateway. WRT init containers, I think most
>>>>>>> users use them just as a workaround. For example, wget a jar from the
>>>>>>> maven repo.
>>>>>>>
>>>>>>> We could implement the functionality in SQL Driver in a more graceful
>>>>>>> way and the flink-supported filesystem approach seems to be a
>>>>>>> good choice.
>>>>>>>
>>>>>>> Also, what do you think about prefixing the config options with
>>>>>>> `sql-driver` instead of just `sql` to be more specific?
>>>>>>>
>>>>>>> LGTM, since SQL Driver is a public interface and the options are
>>>>>>> specific to it.
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>> 2023年6月6日 06:30，Mason Chen <mas.chen6345@gmail.com <ma...@gmail.com> mailto:mas.chen6345@gmail.com <mailto:
>>>>>>> mas.chen6345@gmail.com <ma...@gmail.com> mailto:mas.chen6345@gmail.com>> 写道：
>>>>>>>
>>>>>>> Hi Paul,
>>>>>>>
>>>>>>> +1 for this feature and supporting SQL file + JSON plans. We get a
>>>>>>>
>>>>>>> lot
>>>>>>>
>>>>>>> of
>>>>>>>
>>>>>>> requests to just be able to submit a SQL file, but the JSON plan
>>>>>>> optimizations make sense.
>>>>>>>
>>>>>>> +1 for init containers or a more generalized way of obtaining
>>>>>>>
>>>>>>> arbitrary
>>>>>>>
>>>>>>> files. File fetching isn't specific to just SQL--it also matters for
>>>>>>>
>>>>>>> Java
>>>>>>>
>>>>>>> applications if the user doesn't want to rebuild a Flink image and
>>>>>>>
>>>>>>> just
>>>>>>>
>>>>>>> wants to modify the user application fat jar.
>>>>>>>
>>>>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>>>>>>
>>>>>>> which
>>>>>>>
>>>>>>> should
>>>>>>>
>>>>>>> be required to run Flink in production, so I guess that would be
>>>>>>>
>>>>>>> acceptable
>>>>>>>
>>>>>>> for most
>>>>>>>
>>>>>>> users. WDYT?
>>>>>>>
>>>>>>> If you do go this route, it would be nice to support writing these
>>>>>>>
>>>>>>> files
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> S3/HDFS via Flink. This makes access control and policy management
>>>>>>>
>>>>>>> simpler.
>>>>>>>
>>>>>>> Also, what do you think about prefixing the config options with
>>>>>>> `sql-driver` instead of just `sql` to be more specific?
>>>>>>>
>>>>>>> Best,
>>>>>>> Mason
>>>>>>>
>>>>>>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com
>>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
>>>>>>>
>>>>>>> <mailto:
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>> wrote:
>>>>>>>
>>>>>>> Hi Jark,
>>>>>>>
>>>>>>> Thanks for your input! Please see my comments inline.
>>>>>>>
>>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>>>> DataStream API also doesn't provide a default main class for users,
>>>>>>> why do we need to provide such one for SQL?
>>>>>>>
>>>>>>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>>>>>>>
>>>>>>> submitted
>>>>>>>
>>>>>>> via Flink CLI which actually could be DataStream/Table jobs.
>>>>>>>
>>>>>>> I think a default main class would be user-friendly which eliminates
>>>>>>>
>>>>>>> the
>>>>>>>
>>>>>>> need
>>>>>>> for users to write a main class as SQLRunner in Flink K8s operator
>>>>>>>
>>>>>>> [1].
>>>>>>>
>>>>>>> I thought the proposed SqlDriver was a dedicated main class
>>>>>>>
>>>>>>> accepting
>>>>>>>
>>>>>>> SQL files, is
>>>>>>>
>>>>>>> that correct?
>>>>>>>
>>>>>>> Both JSON plans and SQL files are accepted. SQL Gateway should use
>>>>>>>
>>>>>>> JSON
>>>>>>>
>>>>>>> plans,
>>>>>>> while CLI users may use either JSON plans or SQL files.
>>>>>>>
>>>>>>> Please see the updated FLIP[2] for more details.
>>>>>>>
>>>>>>> Personally, I prefer the way of init containers which doesn't
>>>>>>>
>>>>>>> depend
>>>>>>>
>>>>>>> on
>>>>>>>
>>>>>>> additional components.
>>>>>>> This can reduce the moving parts of a production environment.
>>>>>>> Depending on a distributed file system makes the testing, demo, and
>>>>>>>
>>>>>>> local
>>>>>>>
>>>>>>> setup harder than init containers.
>>>>>>>
>>>>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>>>>>>
>>>>>>> which
>>>>>>>
>>>>>>> should
>>>>>>> be required to run Flink in production, so I guess that would be
>>>>>>> acceptable for most
>>>>>>> users. WDYT?
>>>>>>>
>>>>>>> WRT testing, demo, and local setups, I think we could support the
>>>>>>>
>>>>>>> local
>>>>>>>
>>>>>>> filesystem
>>>>>>> scheme i.e. file://** <file:///**> <file:///**> <> as the state backends do. It works as long as
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> Gateway
>>>>>>> and JobManager(or SQL Driver) can access the resource directory
>>>>>>>
>>>>>>> (specified
>>>>>>>
>>>>>>> via
>>>>>>> `sql-gateway.application.storage-dir`).
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> [1]
>>>>
>>>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.javahttps://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
>>>> <
>>>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.javahttps://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
>>>>
>>>>>>> [2]
>>>>
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>> <
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>
>>>>>>> [3]
>>>>
>>>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>>>> <
>>>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>> 2023年6月3日 12:21，Jark Wu <imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>>
>>>>>>> 写道：
>>>>>>>
>>>>>>> Hi Paul,
>>>>>>>
>>>>>>> Thanks for your reply. I left my comments inline.
>>>>>>>
>>>>>>> As the FLIP said, it’s good to have a default main class for Flink
>>>>>>>
>>>>>>> SQLs,
>>>>>>>
>>>>>>> which allows users to submit Flink SQLs in the same way as
>>>>>>>
>>>>>>> DataStream
>>>>>>>
>>>>>>> jobs, or else users need to write their own main class.
>>>>>>>
>>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>>>> DataStream API also doesn't provide a default main class for users,
>>>>>>> why do we need to provide such one for SQL?
>>>>>>>
>>>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>>>> SessionState? If not, we could make SQL Driver accepts two
>>>>>>>
>>>>>>> serialized
>>>>>>>
>>>>>>> formats:
>>>>>>>
>>>>>>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
>>>>>>>
>>>>>>> the
>>>>>>>
>>>>>>> proposed SqlDriver was a dedicated main class accepting SQL files,
>>>>>>>
>>>>>>> is
>>>>>>>
>>>>>>> that correct?
>>>>>>> If true, we have to ship the SessionState for this case which is a
>>>>>>>
>>>>>>> large
>>>>>>>
>>>>>>> work.
>>>>>>> I think we just need a JsonPlanDriver which is a main class that
>>>>>>>
>>>>>>> accepts
>>>>>>>
>>>>>>> JsonPlan as the parameter.
>>>>>>>
>>>>>>> The common solutions I know is to use distributed file systems or
>>>>>>>
>>>>>>> use
>>>>>>>
>>>>>>> init containers to localize the resources.
>>>>>>>
>>>>>>> Personally, I prefer the way of init containers which doesn't
>>>>>>>
>>>>>>> depend
>>>>>>>
>>>>>>> on
>>>>>>>
>>>>>>> additional components.
>>>>>>> This can reduce the moving parts of a production environment.
>>>>>>> Depending on a distributed file system makes the testing, demo, and
>>>>>>>
>>>>>>> local
>>>>>>>
>>>>>>> setup harder than init containers.
>>>>>>>
>>>>>>> Best,
>>>>>>> Jark
>>>>>>>
>>>>>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com>
>>>>>>>
>>>>>>> <mailto:
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>> <mailto:
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>> wrote:
>>>>>>>
>>>>>>> The FLIP is in the early phase and some details are not included,
>>>>>>>
>>>>>>> but
>>>>>>>
>>>>>>> fortunately, we got lots of valuable ideas from the discussion.
>>>>>>>
>>>>>>> Thanks to everyone who joined the dissuasion!
>>>>>>> @Weihua @Shanmon @Shengkai @Biao @Jark
>>>>>>>
>>>>>>> This weekend I’m gonna revisit and update the FLIP, adding more
>>>>>>> details. Hopefully, we can further align our opinions.
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com> <mailto:
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>> 写道：
>>>>>>>
>>>>>>> Hi Jark,
>>>>>>>
>>>>>>> Thanks a lot for your input!
>>>>>>>
>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>>>>>>
>>>>>>> still
>>>>>>>
>>>>>>> necessary to support SQL Driver?
>>>>>>>
>>>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
>>>>>>>
>>>>>>> simplify
>>>>>>>
>>>>>>> Flink SQL execution with Flink CLI.
>>>>>>>
>>>>>>> As the FLIP said, it’s good to have a default main class for
>>>>>>>
>>>>>>> Flink
>>>>>>>
>>>>>>> SQLs,
>>>>>>>
>>>>>>> which allows users to submit Flink SQLs in the same way as
>>>>>>>
>>>>>>> DataStream
>>>>>>>
>>>>>>> jobs, or else users need to write their own main class.
>>>>>>>
>>>>>>> SQL Driver needs to serialize SessionState which is very
>>>>>>>
>>>>>>> challenging
>>>>>>>
>>>>>>> but not detailed covered in the FLIP.
>>>>>>>
>>>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>>>> SessionState? If not, we could make SQL Driver accepts two
>>>>>>>
>>>>>>> serialized
>>>>>>>
>>>>>>> formats:
>>>>>>>
>>>>>>> - SQL files for user-facing public usage
>>>>>>> - ExecNodeGraph for internal usage
>>>>>>>
>>>>>>> It’s kind of similar to the relationship between job jars and
>>>>>>>
>>>>>>> jobgraphs.
>>>>>>>
>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>>>>>>>
>>>>>>> true?
>>>>>>>
>>>>>>> Is it
>>>>>>>
>>>>>>> possible to support it?
>>>>>>>
>>>>>>> Yes, K8s doesn’t distribute any files. It’s the users’
>>>>>>>
>>>>>>> responsibility
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> make
>>>>>>>
>>>>>>> sure the resources are accessible in the containers. The common
>>>>>>>
>>>>>>> solutions
>>>>>>>
>>>>>>> I know is to use distributed file systems or use init containers
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> localize the
>>>>>>>
>>>>>>> resources.
>>>>>>>
>>>>>>> Now I lean toward introducing a fs to do the distribution job.
>>>>>>>
>>>>>>> WDYT?
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>
>>>>>>> <mailto:
>>>>>>>
>>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>>
>>>>>>>
>>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com> <mailto:
>>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>>>
>>>>>>>
>>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com> <mailto:
>>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>> <mailto:
>>>>>>>
>>>>>>> imjark@gmail.com <ma...@gmail.com> mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com> <mailto:imjark@gmail.commailto:imjark@gmail.com
>>>>>>> <mailto:imjark@gmail.com mailto:imjark@gmail.com>>>>>
>>>>>>>
>>>>>>> 写道：
>>>>>>>
>>>>>>> Hi Paul,
>>>>>>>
>>>>>>> Thanks for starting this discussion. I like the proposal! This
>>>>>>>
>>>>>>> is
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>> frequently requested feature!
>>>>>>>
>>>>>>> I agree with Shengkai that ExecNodeGraph as the submission
>>>>>>>
>>>>>>> object
>>>>>>>
>>>>>>> is a
>>>>>>>
>>>>>>> better idea than SQL file. To be more specific, it should be
>>>>>>>
>>>>>>> JsonPlanGraph
>>>>>>>
>>>>>>> or CompiledPlan which is the serializable representation.
>>>>>>>
>>>>>>> CompiledPlan
>>>>>>>
>>>>>>> is a
>>>>>>>
>>>>>>> clear separation between compiling/optimization/validation and
>>>>>>>
>>>>>>> execution.
>>>>>>>
>>>>>>> This can keep the validation and metadata accessing still on the
>>>>>>>
>>>>>>> SQLGateway
>>>>>>>
>>>>>>> side. This allows SQLGateway to leverage some metadata caching
>>>>>>>
>>>>>>> and
>>>>>>>
>>>>>>> UDF
>>>>>>>
>>>>>>> JAR
>>>>>>>
>>>>>>> caching for better compiling performance.
>>>>>>>
>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>>>>>>
>>>>>>> still
>>>>>>>
>>>>>>> necessary to support SQL Driver? Regarding non-interactive SQL
>>>>>>>
>>>>>>> jobs,
>>>>>>>
>>>>>>> users
>>>>>>>
>>>>>>> can use the Table API program for application mode. SQL Driver
>>>>>>>
>>>>>>> needs
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> serialize SessionState which is very challenging but not
>>>>>>>
>>>>>>> detailed
>>>>>>>
>>>>>>> covered
>>>>>>>
>>>>>>> in the FLIP.
>>>>>>>
>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>>>>>>>
>>>>>>> true?
>>>>>>>
>>>>>>> Is it
>>>>>>>
>>>>>>> possible to support it?
>>>>>>>
>>>>>>> Best,
>>>>>>> Jark
>>>>>>>
>>>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com>
>>>>>>>
>>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
>>>>>>> <mailto:
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>> <mailto:
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>> <mailto:
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>>> wrote:
>>>>>>>
>>>>>>> Hi Weihua,
>>>>>>>
>>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
>>>>>>>
>>>>>>> challenging
>>>>>>>
>>>>>>> parts of this FLIP.
>>>>>>>
>>>>>>> Web submission is not enabled in application mode currently as
>>>>>>>
>>>>>>> you
>>>>>>>
>>>>>>> said,
>>>>>>>
>>>>>>> but it could be changed if we have good reasons.
>>>>>>>
>>>>>>> What do you think about introducing a distributed storage for
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> Gateway?
>>>>>>>
>>>>>>> We could make use of Flink file systems [1] to distribute the
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> Gateway
>>>>>>>
>>>>>>> generated resources, that should solve the problem at its root
>>>>>>>
>>>>>>> cause.
>>>>>>>
>>>>>>> Users could specify Flink-supported file systems to ship files.
>>>>>>>
>>>>>>> It’s
>>>>>>>
>>>>>>> only
>>>>>>>
>>>>>>> required when using SQL Gateway with K8s application mode.
>>>>>>>
>>>>>>> [1]
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>> <
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>> <
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>> <
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>> <
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>> <
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>> <
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>> <
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:
>>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com> <mailto:
>>>>>>>
>>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>> <mailto:
>>>>>>>
>>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com> <mailto:
>>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>>>> 写道：
>>>>>>>
>>>>>>> Thanks Paul for your reply.
>>>>>>>
>>>>>>> SQLDriver looks good to me.
>>>>>>>
>>>>>>> 2. Do you mean a pass the SQL string a configuration or a
>>>>>>>
>>>>>>> program
>>>>>>>
>>>>>>> argument?
>>>>>>>
>>>>>>> I brought this up because we were unable to pass the SQL file
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> Flink
>>>>>>>
>>>>>>> using Kubernetes mode.
>>>>>>> For DataStream/Python users, they need to prepare their images
>>>>>>>
>>>>>>> for
>>>>>>>
>>>>>>> the
>>>>>>>
>>>>>>> jars
>>>>>>>
>>>>>>> and dependencies.
>>>>>>> But for SQL users, they can use a common image to run
>>>>>>>
>>>>>>> different
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> queries
>>>>>>>
>>>>>>> if there are no other udf requirements.
>>>>>>> It would be great if the SQL query and image were not bound.
>>>>>>>
>>>>>>> Using strings is a way to decouple these, but just as you
>>>>>>>
>>>>>>> mentioned,
>>>>>>>
>>>>>>> it's
>>>>>>>
>>>>>>> not easy to pass complex SQL.
>>>>>>>
>>>>>>> use web submission
>>>>>>>
>>>>>>> AFAIK, we can not use web submission in the Application mode.
>>>>>>>
>>>>>>> Please
>>>>>>>
>>>>>>> correct me if I'm wrong.
>>>>>>>
>>>>>>> Best,
>>>>>>> Weihua
>>>>>>>
>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
>>>>>>>
>>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
>>>>>>>
>>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>>
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Biao,
>>>>>>>
>>>>>>> Thanks for your comments!
>>>>>>>
>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
>>>>>>>
>>>>>>> Flink
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> jobs
>>>>>>>
>>>>>>> in
>>>>>>>
>>>>>>> Application mode? More specifically, if we use SQL
>>>>>>>
>>>>>>> client/gateway
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> execute some interactive SQLs like a SELECT query, can we
>>>>>>>
>>>>>>> ask
>>>>>>>
>>>>>>> flink
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> use
>>>>>>>
>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>
>>>>>>> Thanks for pointing it out. I think only DMLs would be
>>>>>>>
>>>>>>> executed
>>>>>>>
>>>>>>> via
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> Driver.
>>>>>>> I'll add the scope to the FLIP.
>>>>>>>
>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>>>
>>>>>>> trivial as
>>>>>>>
>>>>>>> we
>>>>>>>
>>>>>>> can ship files via YARN's tool easily but for K8s, things
>>>>>>>
>>>>>>> can
>>>>>>>
>>>>>>> be
>>>>>>>
>>>>>>> more
>>>>>>>
>>>>>>> complicated as Shengkai said.
>>>>>>>
>>>>>>> Your input is very informative. I’m thinking about using web
>>>>>>>
>>>>>>> submission,
>>>>>>>
>>>>>>> but it requires exposing the JobManager port which could also
>>>>>>>
>>>>>>> be
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>> problem
>>>>>>>
>>>>>>> on K8s.
>>>>>>>
>>>>>>> Another approach is to explicitly require a distributed
>>>>>>>
>>>>>>> storage
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> ship
>>>>>>>
>>>>>>> files,
>>>>>>> but we may need a new deployment executor for that.
>>>>>>>
>>>>>>> What do you think of these two approaches?
>>>>>>>
>>>>>>> 3. Serialization of SessionState: in SessionState, there are
>>>>>>>
>>>>>>> some
>>>>>>>
>>>>>>> unserializable fields
>>>>>>> like
>>>>>>>
>>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>>
>>>>>>> It
>>>>>>>
>>>>>>> may be worthwhile to add more details about the
>>>>>>>
>>>>>>> serialization
>>>>>>>
>>>>>>> part.
>>>>>>>
>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph
>>>>>>>
>>>>>>> as
>>>>>>>
>>>>>>> Shengkai
>>>>>>>
>>>>>>> mentioned, do we eliminate the need for serialization of
>>>>>>>
>>>>>>> SessionState?
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <ma...@gmail.com> mailto:biaogeng7@gmail.com <mailto:
>>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:biaogeng7@gmail.com> <mailto:
>>>>>>>
>>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com>> <mailto:
>>>>>>>
>>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com> <mailto:
>>>>>>> biaogeng7@gmail.com <ma...@gmail.com> mailto:biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com>>>> 写道：
>>>>>>>
>>>>>>> Thanks Paul for the proposal!I believe it would be very
>>>>>>>
>>>>>>> useful
>>>>>>>
>>>>>>> for
>>>>>>>
>>>>>>> flink
>>>>>>>
>>>>>>> users.
>>>>>>> After reading the FLIP, I have some questions:
>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
>>>>>>>
>>>>>>> Flink
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> jobs
>>>>>>>
>>>>>>> in
>>>>>>>
>>>>>>> Application mode? More specifically, if we use SQL
>>>>>>>
>>>>>>> client/gateway
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> execute some interactive SQLs like a SELECT query, can we
>>>>>>>
>>>>>>> ask
>>>>>>>
>>>>>>> flink
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> use
>>>>>>>
>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>>>
>>>>>>> trivial as
>>>>>>>
>>>>>>> we
>>>>>>>
>>>>>>> can ship files via YARN's tool easily but for K8s, things
>>>>>>>
>>>>>>> can
>>>>>>>
>>>>>>> be
>>>>>>>
>>>>>>> more
>>>>>>>
>>>>>>> complicated as Shengkai said. I have implemented a simple
>>>>>>>
>>>>>>> POC
>>>>>>>
>>>>>>> <
>>>>
>>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>> <
>>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>
>>>>>>> <
>>>>
>>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>> <
>>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>
>>>>>>> <
>>>>
>>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>> <
>>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>
>>>>>>> <
>>>>
>>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>> <
>>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>
>>>>>>> based on SQL client before(i.e. consider the SQL client
>>>>>>>
>>>>>>> which
>>>>>>>
>>>>>>> supports
>>>>>>>
>>>>>>> executing a SQL file as the SQL driver in this FLIP). One
>>>>>>>
>>>>>>> problem
>>>>>>>
>>>>>>> I
>>>>>>>
>>>>>>> have
>>>>>>>
>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s
>>>>>>>
>>>>>>> side.
>>>>>>>
>>>>>>> Without
>>>>>>>
>>>>>>> such support, users have to modify the initContainer or
>>>>>>>
>>>>>>> rebuild
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>> new
>>>>>>>
>>>>>>> K8s
>>>>>>>
>>>>>>> image every time to fetch the SQL file. Like the flink k8s
>>>>>>>
>>>>>>> operator,
>>>>>>>
>>>>>>> one
>>>>>>>
>>>>>>> workaround is to utilize the flink config(transforming the
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> file
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>> escaped string like Weihua mentioned) which will be
>>>>>>>
>>>>>>> converted
>>>>>>>
>>>>>>> to a
>>>>>>>
>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger
>>>>>>>
>>>>>>> than
>>>>>>>
>>>>>>> 1MB
>>>>>>>
>>>>>>> <
>>>>>>>
>>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/ <
>>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/>
>>>>>>>
>>>>>>> <
>>>>>>>
>>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/ <
>>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/>> <
>>>>>>>
>>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/ <
>>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/> <
>>>>>>>
>>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/ <
>>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/>>>>).
>>>>>>>
>>>>>>> Not
>>>>>>>
>>>>>>> sure
>>>>>>>
>>>>>>> if we have better solutions.
>>>>>>> 3. Serialization of SessionState: in SessionState, there are
>>>>>>>
>>>>>>> some
>>>>>>>
>>>>>>> unserializable fields
>>>>>>> like
>>>>>>>
>>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>>
>>>>>>> It
>>>>>>>
>>>>>>> may be worthwhile to add more details about the
>>>>>>>
>>>>>>> serialization
>>>>>>>
>>>>>>> part.
>>>>>>>
>>>>>>> Best,
>>>>>>> Biao Geng
>>>>>>>
>>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
>>>>>>> <mailto:
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
>>>>>>>
>>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>>
>>>>>>>
>>>>>>> 于2023年5月31日周三 11:49写道：
>>>>>>>
>>>>>>> Hi Weihua,
>>>>>>>
>>>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>>>
>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
>>>>>>>
>>>>>>> Job.
>>>>>>>
>>>>>>> (Not
>>>>>>>
>>>>>>> strong,
>>>>>>>
>>>>>>> the SQLDriver is fine for me)
>>>>>>>
>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
>>>>>>>
>>>>>>> following
>>>>>>>
>>>>>>> reasons FYI:
>>>>>>>
>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>>>> 2. Flink program's main class is sort of like Driver in
>>>>>>>
>>>>>>> JDBC
>>>>>>>
>>>>>>> which
>>>>>>>
>>>>>>> translates SQLs into
>>>>>>> databases specific languages.
>>>>>>>
>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>>>
>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
>>>>>>>
>>>>>>> we
>>>>>>>
>>>>>>> need
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> prepare
>>>>>>>
>>>>>>> a SQL file in an image for Kubernetes application mode,
>>>>>>>
>>>>>>> which
>>>>>>>
>>>>>>> may
>>>>>>>
>>>>>>> be
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>> bit
>>>>>>>
>>>>>>> cumbersome.
>>>>>>>
>>>>>>> Do you mean a pass the SQL string a configuration or a
>>>>>>>
>>>>>>> program
>>>>>>>
>>>>>>> argument?
>>>>>>>
>>>>>>> I thought it might be convenient for testing propose, but
>>>>>>>
>>>>>>> not
>>>>>>>
>>>>>>> recommended
>>>>>>>
>>>>>>> for production,
>>>>>>> cause Flink SQLs could be complicated and involves lots of
>>>>>>>
>>>>>>> characters
>>>>>>>
>>>>>>> that
>>>>>>>
>>>>>>> need to escape.
>>>>>>>
>>>>>>> WDYT?
>>>>>>>
>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>
>>>>>>> "run-application"
>>>>>>>
>>>>>>> command. Does that mean we need to perform automatic
>>>>>>>
>>>>>>> detection
>>>>>>>
>>>>>>> in
>>>>>>>
>>>>>>> Flink?
>>>>>>>
>>>>>>> Yes! It’s like running a PyFlink job with the following
>>>>>>>
>>>>>>> command:
>>>>>>>
>>>>>>> `./bin/flink run \\ --pyModule table.word_count \\ --pyFiles examples/python/table`
>>>>>>>
>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
>>>>>>>
>>>>>>> Driver
>>>>>>>
>>>>>>> automatically.
>>>>>>>
>>>>>>> [1]
>>>>
>>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>> <
>>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>
>>>>>>> <
>>>>
>>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>> <
>>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>
>>>>>>> <
>>>>
>>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>> <
>>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>
>>>>>>> <
>>>>
>>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>> <
>>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.javahttps://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:
>>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com>
>>>>>>>
>>>>>>> <mailto:
>>>>>>>
>>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>> <mailto:
>>>>>>>
>>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>>> 写道：
>>>>>>>
>>>>>>> Thanks Paul for the proposal.
>>>>>>>
>>>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>>>
>>>>>>> I have a few questions.
>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
>>>>>>>
>>>>>>> Job.
>>>>>>>
>>>>>>> (Not
>>>>>>>
>>>>>>> strong,
>>>>>>>
>>>>>>> the SQLDriver is fine for me)
>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
>>>>>>>
>>>>>>> we
>>>>>>>
>>>>>>> need
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> prepare
>>>>>>>
>>>>>>> a SQL file in an image for Kubernetes application mode,
>>>>>>>
>>>>>>> which
>>>>>>>
>>>>>>> may
>>>>>>>
>>>>>>> be
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>> bit
>>>>>>>
>>>>>>> cumbersome.
>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>
>>>>>>> "run-application"
>>>>>>>
>>>>>>> command. Does that mean we need to perform automatic
>>>>>>>
>>>>>>> detection
>>>>>>>
>>>>>>> in
>>>>>>>
>>>>>>> Flink?
>>>>>>>
>>>>>>> Best,
>>>>>>> Weihua
>>>>>>>
>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>>>>>>>
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
>>>>>>>
>>>>>>> <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
>>>>>>> paullin3280@gmail.com <ma...@gmail.com> mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>>
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi team,
>>>>>>>
>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
>>>>>>>
>>>>>>> introduces
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> driver as the
>>>>>>> default main class for Flink SQL jobs.
>>>>>>>
>>>>>>> Currently, Flink SQL could be executed out of the box
>>>>>>>
>>>>>>> either
>>>>>>>
>>>>>>> via
>>>>>>>
>>>>>>> SQL
>>>>>>>
>>>>>>> Client/Gateway
>>>>>>> or embedded in a Flink Java/Python program.
>>>>>>>
>>>>>>> However, each one has its drawback:
>>>>>>>
>>>>>>> - SQL Client/Gateway doesn’t support the application
>>>>>>>
>>>>>>> deployment
>>>>>>>
>>>>>>> mode
>>>>>>>
>>>>>>> [2]
>>>>>>>
>>>>>>> - Flink Java/Python program requires extra work to write
>>>>>>>
>>>>>>> a
>>>>>>>
>>>>>>> non-SQL
>>>>>>>
>>>>>>> program
>>>>>>>
>>>>>>> Therefore, I propose adding a SQL driver to act as the
>>>>>>>
>>>>>>> default
>>>>>>>
>>>>>>> main
>>>>>>>
>>>>>>> class
>>>>>>>
>>>>>>> for SQL jobs.
>>>>>>> Please see the FLIP docs for details and feel free to
>>>>>>>
>>>>>>> comment.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> [1]
>>>>
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>> <
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>
>>>>>>> <
>>>>
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>> <
>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>
>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541 https://issues.apache.org/jira/browse/FLINK-26541> <
>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541 https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541 https://issues.apache.org/jira/browse/FLINK-26541>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Ferenc Csaky <fe...@pm.me.INVALID>.
Hello devs,
Is any active work happening on this FLIP? As far as I see there
are blockers that needs to happen first to implement regarding
artifact distribution.
Is this work in halt completetly or some efforts are going into
resolve the blockers first or something?
Our platform would benefit this feature a lot, we have a kind of
working custom implementation at the moment, but it is uniquely
adapted to our app and platform.
I could help out to move this forward.
Best,
Ferenc
On Friday, June 30th, 2023 at 04:53, Paul Lam <pa...@gmail.com> wrote:
>
>
> Hi Jing,
>
> Thanks for your input!
>
> > Would you like to add
> > one section to describe(better with script/code example) how to use it in
> > these two scenarios from users' perspective?
>
>
> OK. I’ll update the FLIP with the code snippet after I get the POC branch done.
>
> > NIT: the pictures have transparent background when readers click on it. It
> > would be great if you can replace them with pictures with white background.
>
>
> Fixed. Thanks for pointing that out :)
>
> Best,
> Paul Lam
>
> > 2023年6月27日 06:51，Jing Ge jing@ververica.com.INVALID 写道：
> >
> > Hi Paul,
> >
> > Thanks for driving it and thank you all for the informative discussion! The
> > FLIP is in good shape now. As described in the FLIP, SQL Driver will be
> > mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway in
> > application mode and 2. external system integration. Would you like to add
> > one section to describe(better with script/code example) how to use it in
> > these two scenarios from users' perspective?
> >
> > NIT: the pictures have transparent background when readers click on it. It
> > would be great if you can replace them with pictures with white background.
> >
> > Best regards,
> > Jing
> >
> > On Mon, Jun 26, 2023 at 1:31 PM Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com> wrote:
> >
> > > Hi Shengkai,
> > >
> > > > * How can we ship the json plan to the JobManager?
> > >
> > > The Flink K8s module should be responsible for file distribution. We could
> > > introduce
> > > an option like `kubernetes.storage.dir`. For each flink cluster, there
> > > would be a
> > > dedicated subdirectory, with the pattern like
> > > `${kubernetes.storage.dir}/${cluster-id}`.
> > >
> > > All resources-related options (e.g. pipeline jars, json plans) that are
> > > configured with
> > > scheme `file://` <file://`/> <file:///%60 <file:///%60>> would be uploaded to the resource directory
> > > and downloaded to the
> > > jobmanager, before SQL Driver accesses the files with the original
> > > filenames.
> > >
> > > > * Classloading strategy
> > >
> > > We could directly specify the SQL Gateway jar as the jar file in
> > > PackagedProgram.
> > > It would be treated like a normal user jar and the SQL Driver is loaded
> > > into the user
> > > classloader. WDYT?
> > >
> > > > * Option `$internal.sql-gateway.driver.sql-config` is string type
> > > > I think it's better to use Map type here
> > >
> > > By Map type configuration, do you mean a nested map that contains all
> > > configurations?
> > >
> > > I hope I've explained myself well, it’s a file that contains the extra SQL
> > > configurations, which would be shipped to the jobmanager.
> > >
> > > > * PoC branch
> > >
> > > Sure. I’ll let you know once I get the job done.
> > >
> > > Best,
> > > Paul Lam
> > >
> > > > 2023年6月26日 14:27，Shengkai Fang <fskmine@gmail.com mailto:fskmine@gmail.com> 写道：
> > > >
> > > > Hi, Paul.
> > > >
> > > > Thanks for your update. I have a few questions about the new design:
> > > >
> > > > * How can we ship the json plan to the JobManager?
> > > >
> > > > The current design only exposes an option about the URL of the json
> > > > plan. It seems the gateway is responsible to upload to an external stroage.
> > > > Can we reuse the PipelineOptions.JARS to ship to the remote filesystem?
> > > >
> > > > * Classloading strategy
> > > >
> > > > Currently, the Driver is in the sql-gateway package. It means the Driver
> > > > is not in the JM's classpath directly. Because the sql-gateway jar is now
> > > > in the opt directory rather than lib directory. It may need to add the
> > > > external dependencies as Python does[1]. BTW, I think it's better to move
> > > > the Driver into the flink-table-runtime package, which is much easier to
> > > > find(Sorry for the wrong opinion before).
> > > >
> > > > * Option `$internal.sql-gateway.driver.sql-config` is string type
> > > >
> > > > I think it's better to use Map type here
> > > >
> > > > * PoC branch
> > > >
> > > > Because this FLIP involves many modules, do you have a PoC branch to
> > > > verify it does work?
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > > [1]
> > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
> > > > <
> > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
> > > >
> > > > Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> > > > 于2023年6月19日周一 14:09写道：
> > > > Hi Shengkai,
> > > >
> > > > Sorry for my late reply. It took me some time to update the FLIP.
> > > >
> > > > In the latest FLIP design, SQL Driver is placed in flink-sql-gateway
> > > > module. PTAL.
> > > >
> > > > The FLIP does not cover details about the K8s file distribution, but its
> > > > general usage would
> > > > be very much the same as YARN setups. We could make follow-up
> > > > discussions in the jira
> > > > tickets.
> > > >
> > > > Best,
> > > > Paul Lam
> > > >
> > > > > 2023年6月12日 15:29，Shengkai Fang <fskmine@gmail.com mailto:fskmine@gmail.com <mailto:
> > > > > fskmine@gmail.com mailto:fskmine@gmail.com>> 写道：
> > > > >
> > > > > > If it’s the case, I’m good with introducing a new module and making
> > > > > > SQL Driver
> > > > > > an internal class and accepts JSON plans only.
> > > > >
> > > > > I rethink this again and again. I think it's better to move the
> > > > > SqlDriver into the sql-gateway module because the sql client relies on the
> > > > > sql-gateway to submit the sql and the sql-gateway has the ability to
> > > > > generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
> > > > >
> > > > > * Upload configuration through command line parameter
> > > > >
> > > > > ExecNodeGraph only contains the job's information but it doesn't
> > > > > contain the checkpoint dir, checkpoint interval, execution mode and so on.
> > > > > So I think we should also upload the configuration.
> > > > >
> > > > > * KubernetesClusterDescripter and
> > > > > KubernetesApplicationClusterEntrypoint are responsible for the jar
> > > > > upload/download
> > > > >
> > > > > +1 for the change.
> > > > >
> > > > > Could you update the FLIP about the current discussion?
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > > Yang Wang <wangyang0918@apache.org mailto:wangyang0918@apache.org <mailto:wangyang0918@apache.org mailto:wangyang0918@apache.org>>
> > > > > 于2023年6月12日周一 11:41写道：
> > > > > Sorry for the late reply. I am in favor of introducing such a built-in
> > > > > resource localization mechanism
> > > > > based on Flink FileSystem. Then FLINK-28915[1] could be the second step
> > > > > which will download
> > > > > the jars and dependencies to the JobManager/TaskManager local directory
> > > > > before working.
> > > > >
> > > > > The first step could be done in another ticket in Flink. Or some
> > > > > external
> > > > > Flink jobs management system
> > > > > could also take care of this.
> > > > >
> > > > > [1]. https://issues.apache.org/jira/browse/FLINK-28915 https://issues.apache.org/jira/browse/FLINK-28915 <
> > > > > https://issues.apache.org/jira/browse/FLINK-28915 https://issues.apache.org/jira/browse/FLINK-28915>
> > > > >
> > > > > Best,
> > > > > Yang
> > > > >
> > > > > Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> > > > > 于2023年6月9日周五 17:39写道：
> > > > >
> > > > > > Hi Mason,
> > > > > >
> > > > > > I get your point. I'm increasingly feeling the need to introduce a
> > > > > > built-in
> > > > > > file distribution mechanism for flink-kubernetes module, just like
> > > > > > Spark
> > > > > > does with `spark.kubernetes.file.upload.path` [1].
> > > > > >
> > > > > > I’m assuming the workflow is as follows:
> > > > > >
> > > > > > - KubernetesClusterDescripter uploads all local resources to a remote
> > > > > > storage via Flink filesystem (skips if the resources are already
> > > > > > remote).
> > > > > > - KubernetesApplicationClusterEntrypoint downloads the resources
> > > > > > and put them in the classpath during startup.
> > > > > >
> > > > > > I wouldn't mind splitting it into another FLIP to ensure that
> > > > > > everything is
> > > > > > done correctly.
> > > > > >
> > > > > > cc'ed @Yang to gather more opinions.
> > > > > >
> > > > > > [1]
> > >
> > > https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> > > <
> > > https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > 2023年6月8日 12:15，Mason Chen <mas.chen6345@gmail.com mailto:mas.chen6345@gmail.com <mailto:
> > > > > > mas.chen6345@gmail.com mailto:mas.chen6345@gmail.com>> 写道：
> > > > > >
> > > > > > Hi Paul,
> > > > > >
> > > > > > Thanks for your response!
> > > > > >
> > > > > > I agree that utilizing SQL Drivers in Java applications is equally
> > > > > > important
> > > > > >
> > > > > > as employing them in SQL Gateway. WRT init containers, I think most
> > > > > > users use them just as a workaround. For example, wget a jar from the
> > > > > > maven repo.
> > > > > >
> > > > > > We could implement the functionality in SQL Driver in a more graceful
> > > > > > way and the flink-supported filesystem approach seems to be a
> > > > > > good choice.
> > > > > >
> > > > > > My main point is: can we solve the problem with a design agnostic of
> > > > > > SQL
> > > > > > and Stream API? I mentioned a use case where this ability is useful
> > > > > > for
> > > > > > Java or Stream API applications. Maybe this is even a non-goal to
> > > > > > your FLIP
> > > > > > since you are focusing on the driver entrypoint.
> > > > > >
> > > > > > Jark mentioned some optimizations:
> > > > > >
> > > > > > This allows SQLGateway to leverage some metadata caching and UDF JAR
> > > > > > caching for better compiling performance.
> > > > > >
> > > > > > It would be great to see this even outside the SQLGateway (i.e. UDF
> > > > > > JAR
> > > > > > caching).
> > > > > >
> > > > > > Best,
> > > > > > Mason
> > > > > >
> > > > > > On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskmine@gmail.com mailto:fskmine@gmail.com
> > > > > > <mailto:fskmine@gmail.com mailto:fskmine@gmail.com>> wrote:
> > > > > >
> > > > > > Hi. Paul. Thanks for your update and the update makes me understand
> > > > > > the
> > > > > > design much better.
> > > > > >
> > > > > > But I still have some questions about the FLIP.
> > > > > >
> > > > > > For SQL Gateway, only DMLs need to be delegated to the SQL server
> > > > > > Driver. I would think about the details and update the FLIP. Do you
> > > > > > have
> > > > > >
> > > > > > some
> > > > > >
> > > > > > ideas already?
> > > > > >
> > > > > > If the applicaiton mode can not support library mode, I think we
> > > > > > should
> > > > > > only execute INSERT INTO and UPDATE/ DELETE statement in the
> > > > > > application
> > > > > > mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> > > > > > statements. The ANALYZE TABLE syntax need to register the statistic
> > > > > > to the
> > > > > > catalog after job finishes and the CALL PROCEDURE statement doesn't
> > > > > > generate the ExecNodeGraph.
> > > > > >
> > > > > > * Introduce storage via option `sql-gateway.application.storage-dir`
> > > > > >
> > > > > > If we can not support to submit the jars through web submission, +1 to
> > > > > > introduce the options to upload the files. While I think the uploader
> > > > > > should be responsible to remove the uploaded jars. Can we remove the
> > > > > > jars
> > > > > > if the job is running or gateway exits?
> > > > > >
> > > > > > * JobID is not avaliable
> > > > > >
> > > > > > Can we use the returned rest client by ApplicationDeployer to query
> > > > > > the job
> > > > > > id? I am concerned that users don't know which job is related to the
> > > > > > submitted SQL.
> > > > > >
> > > > > > * Do we need to introduce a new module named flink-table-sql-runner?
> > > > > >
> > > > > > It seems we need to introduce a new module. Will the new module is
> > > > > > available in the distribution package? I agree with Jark that we
> > > > > > don't need
> > > > > > to introduce this for table-API users and these users have their main
> > > > > > class. If we want to make users write the k8s operator more easily, I
> > > > > > think
> > > > > > we should modify the k8s operator repo. If we don't need to support
> > > > > > SQL
> > > > > > files, can we make this jar only visible in the sql-gateway like we
> > > > > > do in
> > > > > > the planner loader?[1]
> > > > > >
> > > > > > [1]
> > >
> > > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> > > <
> > > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> > >
> > > > > > Best,
> > > > > > Shengkai
> > > > > >
> > > > > > Weihua Hu <huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>>
> > > > > > 于2023年6月7日周三 10:52写道：
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Thanks for updating the FLIP.
> > > > > >
> > > > > > I have two cents on the distribution of SQLs and resources.
> > > > > > 1. Should we support a common file distribution mechanism for k8s
> > > > > > application mode?
> > > > > > I have seen some issues and requirements on the mailing list.
> > > > > > In our production environment, we implement the download command in
> > > > > > the
> > > > > > CliFrontend.
> > > > > > And automatically add an init container to the POD for file
> > > > > >
> > > > > > downloading.
> > > > > >
> > > > > > The advantage of this
> > > > > > is that we can use all Flink-supported file systems to store files.
> > > > > >
> > > > > > This need more discussion. I would appreciate hearing more opinions.
> > > > > >
> > > > > > 2. In this FLIP, we distribute files in two different ways in YARN and
> > > > > > Kubernetes. Can we combine it in one way?
> > > > > > If we don't want to implement a common file distribution for k8s
> > > > > > application mode. Could we use the SQLDriver
> > > > > > to download the files both in YARN and K8S? IMO, this can reduce the
> > > > > >
> > > > > > cost
> > > > > >
> > > > > > of code maintenance.
> > > > > >
> > > > > > Best,
> > > > > > Weihua
> > > > > >
> > > > > > On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com
> > > > > > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>> wrote:
> > > > > >
> > > > > > Hi Mason,
> > > > > >
> > > > > > Thanks for your input!
> > > > > >
> > > > > > +1 for init containers or a more generalized way of obtaining
> > > > > >
> > > > > > arbitrary
> > > > > >
> > > > > > files. File fetching isn't specific to just SQL--it also matters for
> > > > > >
> > > > > > Java
> > > > > >
> > > > > > applications if the user doesn't want to rebuild a Flink image and
> > > > > >
> > > > > > just
> > > > > >
> > > > > > wants to modify the user application fat jar.
> > > > > >
> > > > > > I agree that utilizing SQL Drivers in Java applications is equally
> > > > > > important
> > > > > > as employing them in SQL Gateway. WRT init containers, I think most
> > > > > > users use them just as a workaround. For example, wget a jar from the
> > > > > > maven repo.
> > > > > >
> > > > > > We could implement the functionality in SQL Driver in a more graceful
> > > > > > way and the flink-supported filesystem approach seems to be a
> > > > > > good choice.
> > > > > >
> > > > > > Also, what do you think about prefixing the config options with
> > > > > > `sql-driver` instead of just `sql` to be more specific?
> > > > > >
> > > > > > LGTM, since SQL Driver is a public interface and the options are
> > > > > > specific to it.
> > > > > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > 2023年6月6日 06:30，Mason Chen <mas.chen6345@gmail.com mailto:mas.chen6345@gmail.com <mailto:
> > > > > > mas.chen6345@gmail.com mailto:mas.chen6345@gmail.com>> 写道：
> > > > > >
> > > > > > Hi Paul,
> > > > > >
> > > > > > +1 for this feature and supporting SQL file + JSON plans. We get a
> > > > > >
> > > > > > lot
> > > > > >
> > > > > > of
> > > > > >
> > > > > > requests to just be able to submit a SQL file, but the JSON plan
> > > > > > optimizations make sense.
> > > > > >
> > > > > > +1 for init containers or a more generalized way of obtaining
> > > > > >
> > > > > > arbitrary
> > > > > >
> > > > > > files. File fetching isn't specific to just SQL--it also matters for
> > > > > >
> > > > > > Java
> > > > > >
> > > > > > applications if the user doesn't want to rebuild a Flink image and
> > > > > >
> > > > > > just
> > > > > >
> > > > > > wants to modify the user application fat jar.
> > > > > >
> > > > > > Please note that we could reuse the checkpoint storage like S3/HDFS,
> > > > > >
> > > > > > which
> > > > > >
> > > > > > should
> > > > > >
> > > > > > be required to run Flink in production, so I guess that would be
> > > > > >
> > > > > > acceptable
> > > > > >
> > > > > > for most
> > > > > >
> > > > > > users. WDYT?
> > > > > >
> > > > > > If you do go this route, it would be nice to support writing these
> > > > > >
> > > > > > files
> > > > > >
> > > > > > to
> > > > > >
> > > > > > S3/HDFS via Flink. This makes access control and policy management
> > > > > >
> > > > > > simpler.
> > > > > >
> > > > > > Also, what do you think about prefixing the config options with
> > > > > > `sql-driver` instead of just `sql` to be more specific?
> > > > > >
> > > > > > Best,
> > > > > > Mason
> > > > > >
> > > > > > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com
> > > > > > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
> > > > > >
> > > > > > <mailto:
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>> wrote:
> > > > > >
> > > > > > Hi Jark,
> > > > > >
> > > > > > Thanks for your input! Please see my comments inline.
> > > > > >
> > > > > > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > > > > > DataStream API also doesn't provide a default main class for users,
> > > > > > why do we need to provide such one for SQL?
> > > > > >
> > > > > > Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> > > > > >
> > > > > > submitted
> > > > > >
> > > > > > via Flink CLI which actually could be DataStream/Table jobs.
> > > > > >
> > > > > > I think a default main class would be user-friendly which eliminates
> > > > > >
> > > > > > the
> > > > > >
> > > > > > need
> > > > > > for users to write a main class as SQLRunner in Flink K8s operator
> > > > > >
> > > > > > [1].
> > > > > >
> > > > > > I thought the proposed SqlDriver was a dedicated main class
> > > > > >
> > > > > > accepting
> > > > > >
> > > > > > SQL files, is
> > > > > >
> > > > > > that correct?
> > > > > >
> > > > > > Both JSON plans and SQL files are accepted. SQL Gateway should use
> > > > > >
> > > > > > JSON
> > > > > >
> > > > > > plans,
> > > > > > while CLI users may use either JSON plans or SQL files.
> > > > > >
> > > > > > Please see the updated FLIP[2] for more details.
> > > > > >
> > > > > > Personally, I prefer the way of init containers which doesn't
> > > > > >
> > > > > > depend
> > > > > >
> > > > > > on
> > > > > >
> > > > > > additional components.
> > > > > > This can reduce the moving parts of a production environment.
> > > > > > Depending on a distributed file system makes the testing, demo, and
> > > > > >
> > > > > > local
> > > > > >
> > > > > > setup harder than init containers.
> > > > > >
> > > > > > Please note that we could reuse the checkpoint storage like S3/HDFS,
> > > > > >
> > > > > > which
> > > > > >
> > > > > > should
> > > > > > be required to run Flink in production, so I guess that would be
> > > > > > acceptable for most
> > > > > > users. WDYT?
> > > > > >
> > > > > > WRT testing, demo, and local setups, I think we could support the
> > > > > >
> > > > > > local
> > > > > >
> > > > > > filesystem
> > > > > > scheme i.e. file://** <file:///**> <> as the state backends do. It works as long as
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > Gateway
> > > > > > and JobManager(or SQL Driver) can access the resource directory
> > > > > >
> > > > > > (specified
> > > > > >
> > > > > > via
> > > > > > `sql-gateway.application.storage-dir`).
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > [1]
> > >
> > > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> > > <
> > > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> > >
> > > > > > [2]
> > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > > <
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > >
> > > > > > [3]
> > >
> > > https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> > > <
> > > https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > 2023年6月3日 12:21，Jark Wu <imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>>
> > > > > > 写道：
> > > > > >
> > > > > > Hi Paul,
> > > > > >
> > > > > > Thanks for your reply. I left my comments inline.
> > > > > >
> > > > > > As the FLIP said, it’s good to have a default main class for Flink
> > > > > >
> > > > > > SQLs,
> > > > > >
> > > > > > which allows users to submit Flink SQLs in the same way as
> > > > > >
> > > > > > DataStream
> > > > > >
> > > > > > jobs, or else users need to write their own main class.
> > > > > >
> > > > > > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > > > > > DataStream API also doesn't provide a default main class for users,
> > > > > > why do we need to provide such one for SQL?
> > > > > >
> > > > > > With the help of ExecNodeGraph, do we still need the serialized
> > > > > > SessionState? If not, we could make SQL Driver accepts two
> > > > > >
> > > > > > serialized
> > > > > >
> > > > > > formats:
> > > > > >
> > > > > > No, ExecNodeGraph doesn't need to serialize SessionState. I thought
> > > > > >
> > > > > > the
> > > > > >
> > > > > > proposed SqlDriver was a dedicated main class accepting SQL files,
> > > > > >
> > > > > > is
> > > > > >
> > > > > > that correct?
> > > > > > If true, we have to ship the SessionState for this case which is a
> > > > > >
> > > > > > large
> > > > > >
> > > > > > work.
> > > > > > I think we just need a JsonPlanDriver which is a main class that
> > > > > >
> > > > > > accepts
> > > > > >
> > > > > > JsonPlan as the parameter.
> > > > > >
> > > > > > The common solutions I know is to use distributed file systems or
> > > > > >
> > > > > > use
> > > > > >
> > > > > > init containers to localize the resources.
> > > > > >
> > > > > > Personally, I prefer the way of init containers which doesn't
> > > > > >
> > > > > > depend
> > > > > >
> > > > > > on
> > > > > >
> > > > > > additional components.
> > > > > > This can reduce the moving parts of a production environment.
> > > > > > Depending on a distributed file system makes the testing, demo, and
> > > > > >
> > > > > > local
> > > > > >
> > > > > > setup harder than init containers.
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > >
> > > > > > On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com>
> > > > > >
> > > > > > <mailto:
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>> <mailto:
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>> wrote:
> > > > > >
> > > > > > The FLIP is in the early phase and some details are not included,
> > > > > >
> > > > > > but
> > > > > >
> > > > > > fortunately, we got lots of valuable ideas from the discussion.
> > > > > >
> > > > > > Thanks to everyone who joined the dissuasion!
> > > > > > @Weihua @Shanmon @Shengkai @Biao @Jark
> > > > > >
> > > > > > This weekend I’m gonna revisit and update the FLIP, adding more
> > > > > > details. Hopefully, we can further align our opinions.
> > > > > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>> 写道：
> > > > > >
> > > > > > Hi Jark,
> > > > > >
> > > > > > Thanks a lot for your input!
> > > > > >
> > > > > > If we decide to submit ExecNodeGraph instead of SQL file, is it
> > > > > >
> > > > > > still
> > > > > >
> > > > > > necessary to support SQL Driver?
> > > > > >
> > > > > > I think so. Apart from usage in SQL Gateway, SQL Driver could
> > > > > >
> > > > > > simplify
> > > > > >
> > > > > > Flink SQL execution with Flink CLI.
> > > > > >
> > > > > > As the FLIP said, it’s good to have a default main class for
> > > > > >
> > > > > > Flink
> > > > > >
> > > > > > SQLs,
> > > > > >
> > > > > > which allows users to submit Flink SQLs in the same way as
> > > > > >
> > > > > > DataStream
> > > > > >
> > > > > > jobs, or else users need to write their own main class.
> > > > > >
> > > > > > SQL Driver needs to serialize SessionState which is very
> > > > > >
> > > > > > challenging
> > > > > >
> > > > > > but not detailed covered in the FLIP.
> > > > > >
> > > > > > With the help of ExecNodeGraph, do we still need the serialized
> > > > > > SessionState? If not, we could make SQL Driver accepts two
> > > > > >
> > > > > > serialized
> > > > > >
> > > > > > formats:
> > > > > >
> > > > > > - SQL files for user-facing public usage
> > > > > > - ExecNodeGraph for internal usage
> > > > > >
> > > > > > It’s kind of similar to the relationship between job jars and
> > > > > >
> > > > > > jobgraphs.
> > > > > >
> > > > > > Regarding "K8S doesn't support shipping multiple jars", is that
> > > > > >
> > > > > > true?
> > > > > >
> > > > > > Is it
> > > > > >
> > > > > > possible to support it?
> > > > > >
> > > > > > Yes, K8s doesn’t distribute any files. It’s the users’
> > > > > >
> > > > > > responsibility
> > > > > >
> > > > > > to
> > > > > >
> > > > > > make
> > > > > >
> > > > > > sure the resources are accessible in the containers. The common
> > > > > >
> > > > > > solutions
> > > > > >
> > > > > > I know is to use distributed file systems or use init containers
> > > > > >
> > > > > > to
> > > > > >
> > > > > > localize the
> > > > > >
> > > > > > resources.
> > > > > >
> > > > > > Now I lean toward introducing a fs to do the distribution job.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > 2023年6月1日 20:33，Jark Wu <imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>
> > > > > > <mailto:
> > > > > >
> > > > > > imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>>
> > > > > >
> > > > > > <mailto:imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com> <mailto:
> > > > > > imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>>>
> > > > > >
> > > > > > <mailto:imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com> <mailto:
> > > > > > imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com>> <mailto:
> > > > > >
> > > > > > imjark@gmail.com mailto:imjark@gmail.com <mailto:imjark@gmail.com mailto:imjark@gmail.com> <mailto:imjark@gmail.com mailto:imjark@gmail.com
> > > > > > <mailto:imjark@gmail.com mailto:imjark@gmail.com>>>>>
> > > > > >
> > > > > > 写道：
> > > > > >
> > > > > > Hi Paul,
> > > > > >
> > > > > > Thanks for starting this discussion. I like the proposal! This
> > > > > >
> > > > > > is
> > > > > >
> > > > > > a
> > > > > >
> > > > > > frequently requested feature!
> > > > > >
> > > > > > I agree with Shengkai that ExecNodeGraph as the submission
> > > > > >
> > > > > > object
> > > > > >
> > > > > > is a
> > > > > >
> > > > > > better idea than SQL file. To be more specific, it should be
> > > > > >
> > > > > > JsonPlanGraph
> > > > > >
> > > > > > or CompiledPlan which is the serializable representation.
> > > > > >
> > > > > > CompiledPlan
> > > > > >
> > > > > > is a
> > > > > >
> > > > > > clear separation between compiling/optimization/validation and
> > > > > >
> > > > > > execution.
> > > > > >
> > > > > > This can keep the validation and metadata accessing still on the
> > > > > >
> > > > > > SQLGateway
> > > > > >
> > > > > > side. This allows SQLGateway to leverage some metadata caching
> > > > > >
> > > > > > and
> > > > > >
> > > > > > UDF
> > > > > >
> > > > > > JAR
> > > > > >
> > > > > > caching for better compiling performance.
> > > > > >
> > > > > > If we decide to submit ExecNodeGraph instead of SQL file, is it
> > > > > >
> > > > > > still
> > > > > >
> > > > > > necessary to support SQL Driver? Regarding non-interactive SQL
> > > > > >
> > > > > > jobs,
> > > > > >
> > > > > > users
> > > > > >
> > > > > > can use the Table API program for application mode. SQL Driver
> > > > > >
> > > > > > needs
> > > > > >
> > > > > > to
> > > > > >
> > > > > > serialize SessionState which is very challenging but not
> > > > > >
> > > > > > detailed
> > > > > >
> > > > > > covered
> > > > > >
> > > > > > in the FLIP.
> > > > > >
> > > > > > Regarding "K8S doesn't support shipping multiple jars", is that
> > > > > >
> > > > > > true?
> > > > > >
> > > > > > Is it
> > > > > >
> > > > > > possible to support it?
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > >
> > > > > > On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com>
> > > > > >
> > > > > > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> > > > > > <mailto:
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>> <mailto:
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>> <mailto:
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>>> wrote:
> > > > > >
> > > > > > Hi Weihua,
> > > > > >
> > > > > > You’re right. Distributing the SQLs to the TMs is one of the
> > > > > >
> > > > > > challenging
> > > > > >
> > > > > > parts of this FLIP.
> > > > > >
> > > > > > Web submission is not enabled in application mode currently as
> > > > > >
> > > > > > you
> > > > > >
> > > > > > said,
> > > > > >
> > > > > > but it could be changed if we have good reasons.
> > > > > >
> > > > > > What do you think about introducing a distributed storage for
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > Gateway?
> > > > > >
> > > > > > We could make use of Flink file systems [1] to distribute the
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > Gateway
> > > > > >
> > > > > > generated resources, that should solve the problem at its root
> > > > > >
> > > > > > cause.
> > > > > >
> > > > > > Users could specify Flink-supported file systems to ship files.
> > > > > >
> > > > > > It’s
> > > > > >
> > > > > > only
> > > > > >
> > > > > > required when using SQL Gateway with K8s application mode.
> > > > > >
> > > > > > [1]
> > >
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > > > > > <
> > >
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > > > > > <
> > >
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > > > > > <
> > >
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > > > > > <
> > >
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > > > > > <
> > >
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > > > > > <
> > >
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > > > > > <
> > >
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com <mailto:
> > > > > > huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com> <mailto:
> > > > > >
> > > > > > huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>> <mailto:
> > > > > >
> > > > > > huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com> <mailto:
> > > > > > huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>>>> 写道：
> > > > > >
> > > > > > Thanks Paul for your reply.
> > > > > >
> > > > > > SQLDriver looks good to me.
> > > > > >
> > > > > > 2. Do you mean a pass the SQL string a configuration or a
> > > > > >
> > > > > > program
> > > > > >
> > > > > > argument?
> > > > > >
> > > > > > I brought this up because we were unable to pass the SQL file
> > > > > >
> > > > > > to
> > > > > >
> > > > > > Flink
> > > > > >
> > > > > > using Kubernetes mode.
> > > > > > For DataStream/Python users, they need to prepare their images
> > > > > >
> > > > > > for
> > > > > >
> > > > > > the
> > > > > >
> > > > > > jars
> > > > > >
> > > > > > and dependencies.
> > > > > > But for SQL users, they can use a common image to run
> > > > > >
> > > > > > different
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > queries
> > > > > >
> > > > > > if there are no other udf requirements.
> > > > > > It would be great if the SQL query and image were not bound.
> > > > > >
> > > > > > Using strings is a way to decouple these, but just as you
> > > > > >
> > > > > > mentioned,
> > > > > >
> > > > > > it's
> > > > > >
> > > > > > not easy to pass complex SQL.
> > > > > >
> > > > > > use web submission
> > > > > >
> > > > > > AFAIK, we can not use web submission in the Application mode.
> > > > > >
> > > > > > Please
> > > > > >
> > > > > > correct me if I'm wrong.
> > > > > >
> > > > > > Best,
> > > > > > Weihua
> > > > > >
> > > > > > On Wed, May 31, 2023 at 9:37 PM Paul Lam <
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
> > > > > >
> > > > > > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>
> > > > > >
> > > > > > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi Biao,
> > > > > >
> > > > > > Thanks for your comments!
> > > > > >
> > > > > > 1. Scope: is this FLIP only targeted for non-interactive
> > > > > >
> > > > > > Flink
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > jobs
> > > > > >
> > > > > > in
> > > > > >
> > > > > > Application mode? More specifically, if we use SQL
> > > > > >
> > > > > > client/gateway
> > > > > >
> > > > > > to
> > > > > >
> > > > > > execute some interactive SQLs like a SELECT query, can we
> > > > > >
> > > > > > ask
> > > > > >
> > > > > > flink
> > > > > >
> > > > > > to
> > > > > >
> > > > > > use
> > > > > >
> > > > > > Application mode to execute those queries after this FLIP?
> > > > > >
> > > > > > Thanks for pointing it out. I think only DMLs would be
> > > > > >
> > > > > > executed
> > > > > >
> > > > > > via
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > Driver.
> > > > > > I'll add the scope to the FLIP.
> > > > > >
> > > > > > 2. Deployment: I believe in YARN mode, the implementation is
> > > > > >
> > > > > > trivial as
> > > > > >
> > > > > > we
> > > > > >
> > > > > > can ship files via YARN's tool easily but for K8s, things
> > > > > >
> > > > > > can
> > > > > >
> > > > > > be
> > > > > >
> > > > > > more
> > > > > >
> > > > > > complicated as Shengkai said.
> > > > > >
> > > > > > Your input is very informative. I’m thinking about using web
> > > > > >
> > > > > > submission,
> > > > > >
> > > > > > but it requires exposing the JobManager port which could also
> > > > > >
> > > > > > be
> > > > > >
> > > > > > a
> > > > > >
> > > > > > problem
> > > > > >
> > > > > > on K8s.
> > > > > >
> > > > > > Another approach is to explicitly require a distributed
> > > > > >
> > > > > > storage
> > > > > >
> > > > > > to
> > > > > >
> > > > > > ship
> > > > > >
> > > > > > files,
> > > > > > but we may need a new deployment executor for that.
> > > > > >
> > > > > > What do you think of these two approaches?
> > > > > >
> > > > > > 3. Serialization of SessionState: in SessionState, there are
> > > > > >
> > > > > > some
> > > > > >
> > > > > > unserializable fields
> > > > > > like
> > > > > >
> > > > > > org.apache.flink.table.resource.ResourceManager#userClassLoader.
> > > > > >
> > > > > > It
> > > > > >
> > > > > > may be worthwhile to add more details about the
> > > > > >
> > > > > > serialization
> > > > > >
> > > > > > part.
> > > > > >
> > > > > > I agree. That’s a missing part. But if we use ExecNodeGraph
> > > > > >
> > > > > > as
> > > > > >
> > > > > > Shengkai
> > > > > >
> > > > > > mentioned, do we eliminate the need for serialization of
> > > > > >
> > > > > > SessionState?
> > > > > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com mailto:biaogeng7@gmail.com <mailto:
> > > > > > biaogeng7@gmail.com mailto:biaogeng7@gmail.com> <mailto:
> > > > > >
> > > > > > biaogeng7@gmail.com mailto:biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com>> <mailto:
> > > > > >
> > > > > > biaogeng7@gmail.com mailto:biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com> <mailto:
> > > > > > biaogeng7@gmail.com mailto:biaogeng7@gmail.com <mailto:biaogeng7@gmail.com mailto:biaogeng7@gmail.com>>>> 写道：
> > > > > >
> > > > > > Thanks Paul for the proposal!I believe it would be very
> > > > > >
> > > > > > useful
> > > > > >
> > > > > > for
> > > > > >
> > > > > > flink
> > > > > >
> > > > > > users.
> > > > > > After reading the FLIP, I have some questions:
> > > > > > 1. Scope: is this FLIP only targeted for non-interactive
> > > > > >
> > > > > > Flink
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > jobs
> > > > > >
> > > > > > in
> > > > > >
> > > > > > Application mode? More specifically, if we use SQL
> > > > > >
> > > > > > client/gateway
> > > > > >
> > > > > > to
> > > > > >
> > > > > > execute some interactive SQLs like a SELECT query, can we
> > > > > >
> > > > > > ask
> > > > > >
> > > > > > flink
> > > > > >
> > > > > > to
> > > > > >
> > > > > > use
> > > > > >
> > > > > > Application mode to execute those queries after this FLIP?
> > > > > > 2. Deployment: I believe in YARN mode, the implementation is
> > > > > >
> > > > > > trivial as
> > > > > >
> > > > > > we
> > > > > >
> > > > > > can ship files via YARN's tool easily but for K8s, things
> > > > > >
> > > > > > can
> > > > > >
> > > > > > be
> > > > > >
> > > > > > more
> > > > > >
> > > > > > complicated as Shengkai said. I have implemented a simple
> > > > > >
> > > > > > POC
> > > > > >
> > > > > > <
> > >
> > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > > <
> > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > >
> > > > > > <
> > >
> > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > > <
> > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > >
> > > > > > <
> > >
> > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > > <
> > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > >
> > > > > > <
> > >
> > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > > <
> > > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > >
> > > > > > based on SQL client before(i.e. consider the SQL client
> > > > > >
> > > > > > which
> > > > > >
> > > > > > supports
> > > > > >
> > > > > > executing a SQL file as the SQL driver in this FLIP). One
> > > > > >
> > > > > > problem
> > > > > >
> > > > > > I
> > > > > >
> > > > > > have
> > > > > >
> > > > > > met is how do we ship SQL files ( or Job Graph) to the k8s
> > > > > >
> > > > > > side.
> > > > > >
> > > > > > Without
> > > > > >
> > > > > > such support, users have to modify the initContainer or
> > > > > >
> > > > > > rebuild
> > > > > >
> > > > > > a
> > > > > >
> > > > > > new
> > > > > >
> > > > > > K8s
> > > > > >
> > > > > > image every time to fetch the SQL file. Like the flink k8s
> > > > > >
> > > > > > operator,
> > > > > >
> > > > > > one
> > > > > >
> > > > > > workaround is to utilize the flink config(transforming the
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > file
> > > > > >
> > > > > > to
> > > > > >
> > > > > > a
> > > > > >
> > > > > > escaped string like Weihua mentioned) which will be
> > > > > >
> > > > > > converted
> > > > > >
> > > > > > to a
> > > > > >
> > > > > > ConfigMap but K8s has size limit of ConfigMaps(no larger
> > > > > >
> > > > > > than
> > > > > >
> > > > > > 1MB
> > > > > >
> > > > > > <
> > > > > >
> > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/ <
> > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/>
> > > > > >
> > > > > > <
> > > > > >
> > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/ <
> > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/>> <
> > > > > >
> > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/ <
> > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/> <
> > > > > >
> > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/ <
> > > > > > https://kubernetes.io/docs/concepts/configuration/configmap/ https://kubernetes.io/docs/concepts/configuration/configmap/>>>>).
> > > > > >
> > > > > > Not
> > > > > >
> > > > > > sure
> > > > > >
> > > > > > if we have better solutions.
> > > > > > 3. Serialization of SessionState: in SessionState, there are
> > > > > >
> > > > > > some
> > > > > >
> > > > > > unserializable fields
> > > > > > like
> > > > > >
> > > > > > org.apache.flink.table.resource.ResourceManager#userClassLoader.
> > > > > >
> > > > > > It
> > > > > >
> > > > > > may be worthwhile to add more details about the
> > > > > >
> > > > > > serialization
> > > > > >
> > > > > > part.
> > > > > >
> > > > > > Best,
> > > > > > Biao Geng
> > > > > >
> > > > > > Paul Lam <paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
> > > > > > <mailto:
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
> > > > > >
> > > > > > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>>
> > > > > >
> > > > > > 于2023年5月31日周三 11:49写道：
> > > > > >
> > > > > > Hi Weihua,
> > > > > >
> > > > > > Thanks a lot for your input! Please see my comments inline.
> > > > > >
> > > > > > - Is SQLRunner the better name? We use this to run a SQL
> > > > > >
> > > > > > Job.
> > > > > >
> > > > > > (Not
> > > > > >
> > > > > > strong,
> > > > > >
> > > > > > the SQLDriver is fine for me)
> > > > > >
> > > > > > I’ve thought about SQL Runner but picked SQL Driver for the
> > > > > >
> > > > > > following
> > > > > >
> > > > > > reasons FYI:
> > > > > >
> > > > > > 1. I have a PythonDriver doing the same job for PyFlink [1]
> > > > > > 2. Flink program's main class is sort of like Driver in
> > > > > >
> > > > > > JDBC
> > > > > >
> > > > > > which
> > > > > >
> > > > > > translates SQLs into
> > > > > > databases specific languages.
> > > > > >
> > > > > > In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> > > > > >
> > > > > > - Could we run SQL jobs using SQL in strings? Otherwise,
> > > > > >
> > > > > > we
> > > > > >
> > > > > > need
> > > > > >
> > > > > > to
> > > > > >
> > > > > > prepare
> > > > > >
> > > > > > a SQL file in an image for Kubernetes application mode,
> > > > > >
> > > > > > which
> > > > > >
> > > > > > may
> > > > > >
> > > > > > be
> > > > > >
> > > > > > a
> > > > > >
> > > > > > bit
> > > > > >
> > > > > > cumbersome.
> > > > > >
> > > > > > Do you mean a pass the SQL string a configuration or a
> > > > > >
> > > > > > program
> > > > > >
> > > > > > argument?
> > > > > >
> > > > > > I thought it might be convenient for testing propose, but
> > > > > >
> > > > > > not
> > > > > >
> > > > > > recommended
> > > > > >
> > > > > > for production,
> > > > > > cause Flink SQLs could be complicated and involves lots of
> > > > > >
> > > > > > characters
> > > > > >
> > > > > > that
> > > > > >
> > > > > > need to escape.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > - I noticed that we don't specify the SQLDriver jar in the
> > > > > >
> > > > > > "run-application"
> > > > > >
> > > > > > command. Does that mean we need to perform automatic
> > > > > >
> > > > > > detection
> > > > > >
> > > > > > in
> > > > > >
> > > > > > Flink?
> > > > > >
> > > > > > Yes! It’s like running a PyFlink job with the following
> > > > > >
> > > > > > command:
> > > > > >
> > > > > > `./bin/flink run \\ --pyModule table.word_count \\ --pyFiles examples/python/table`
> > > > > >
> > > > > > The CLI determines if it’s a SQL job, if yes apply the SQL
> > > > > >
> > > > > > Driver
> > > > > >
> > > > > > automatically.
> > > > > >
> > > > > > [1]
> > >
> > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > > <
> > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > >
> > > > > > <
> > >
> > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > > <
> > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > >
> > > > > > <
> > >
> > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > > <
> > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > >
> > > > > > <
> > >
> > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > > <
> > > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > >
> > > > > > Best,
> > > > > > Paul Lam
> > > > > >
> > > > > > 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com <mailto:
> > > > > > huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>
> > > > > >
> > > > > > <mailto:
> > > > > >
> > > > > > huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>> <mailto:
> > > > > >
> > > > > > huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com <mailto:huweihua.ckl@gmail.com mailto:huweihua.ckl@gmail.com>>> 写道：
> > > > > >
> > > > > > Thanks Paul for the proposal.
> > > > > >
> > > > > > +1 for this. It is valuable in improving ease of use.
> > > > > >
> > > > > > I have a few questions.
> > > > > > - Is SQLRunner the better name? We use this to run a SQL
> > > > > >
> > > > > > Job.
> > > > > >
> > > > > > (Not
> > > > > >
> > > > > > strong,
> > > > > >
> > > > > > the SQLDriver is fine for me)
> > > > > > - Could we run SQL jobs using SQL in strings? Otherwise,
> > > > > >
> > > > > > we
> > > > > >
> > > > > > need
> > > > > >
> > > > > > to
> > > > > >
> > > > > > prepare
> > > > > >
> > > > > > a SQL file in an image for Kubernetes application mode,
> > > > > >
> > > > > > which
> > > > > >
> > > > > > may
> > > > > >
> > > > > > be
> > > > > >
> > > > > > a
> > > > > >
> > > > > > bit
> > > > > >
> > > > > > cumbersome.
> > > > > > - I noticed that we don't specify the SQLDriver jar in the
> > > > > >
> > > > > > "run-application"
> > > > > >
> > > > > > command. Does that mean we need to perform automatic
> > > > > >
> > > > > > detection
> > > > > >
> > > > > > in
> > > > > >
> > > > > > Flink?
> > > > > >
> > > > > > Best,
> > > > > > Weihua
> > > > > >
> > > > > > On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> > > > > >
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>
> > > > > >
> > > > > > <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com> <mailto:
> > > > > > paullin3280@gmail.com mailto:paullin3280@gmail.com <mailto:paullin3280@gmail.com mailto:paullin3280@gmail.com>>>>
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > I’d like to start a discussion about FLIP-316 [1], which
> > > > > >
> > > > > > introduces
> > > > > >
> > > > > > a
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > driver as the
> > > > > > default main class for Flink SQL jobs.
> > > > > >
> > > > > > Currently, Flink SQL could be executed out of the box
> > > > > >
> > > > > > either
> > > > > >
> > > > > > via
> > > > > >
> > > > > > SQL
> > > > > >
> > > > > > Client/Gateway
> > > > > > or embedded in a Flink Java/Python program.
> > > > > >
> > > > > > However, each one has its drawback:
> > > > > >
> > > > > > - SQL Client/Gateway doesn’t support the application
> > > > > >
> > > > > > deployment
> > > > > >
> > > > > > mode
> > > > > >
> > > > > > [2]
> > > > > >
> > > > > > - Flink Java/Python program requires extra work to write
> > > > > >
> > > > > > a
> > > > > >
> > > > > > non-SQL
> > > > > >
> > > > > > program
> > > > > >
> > > > > > Therefore, I propose adding a SQL driver to act as the
> > > > > >
> > > > > > default
> > > > > >
> > > > > > main
> > > > > >
> > > > > > class
> > > > > >
> > > > > > for SQL jobs.
> > > > > > Please see the FLIP docs for details and feel free to
> > > > > >
> > > > > > comment.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > [1]
> > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> > > <
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> > >
> > > > > > <
> > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > > <
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > >
> > > > > > [2] https://issues.apache.org/jira/browse/FLINK-26541 https://issues.apache.org/jira/browse/FLINK-26541 <
> > > > > > https://issues.apache.org/jira/browse/FLINK-26541 https://issues.apache.org/jira/browse/FLINK-26541> <
> > > > > > https://issues.apache.org/jira/browse/FLINK-26541 https://issues.apache.org/jira/browse/FLINK-26541 <
> > > > > > https://issues.apache.org/jira/browse/FLINK-26541 https://issues.apache.org/jira/browse/FLINK-26541>>
> > > > > >
> > > > > > Best,
> > > > > > Paul Lam
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Jing,
Thanks for your input!
> Would you like to add
> one section to describe(better with script/code example) how to use it in
> these two scenarios from users' perspective?
OK. I’ll update the FLIP with the code snippet after I get the POC branch done.
> NIT: the pictures have transparent background when readers click on it. It
> would be great if you can replace them with pictures with white background.
Fixed. Thanks for pointing that out :)
Best,
Paul Lam
> 2023年6月27日 06:51，Jing Ge <ji...@ververica.com.INVALID> 写道：
>
> Hi Paul,
>
> Thanks for driving it and thank you all for the informative discussion! The
> FLIP is in good shape now. As described in the FLIP, SQL Driver will be
> mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway in
> application mode and 2. external system integration. Would you like to add
> one section to describe(better with script/code example) how to use it in
> these two scenarios from users' perspective?
>
> NIT: the pictures have transparent background when readers click on it. It
> would be great if you can replace them with pictures with white background.
>
> Best regards,
> Jing
>
> On Mon, Jun 26, 2023 at 1:31 PM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
>
>> Hi Shengkai,
>>
>>> * How can we ship the json plan to the JobManager?
>>
>> The Flink K8s module should be responsible for file distribution. We could
>> introduce
>> an option like `kubernetes.storage.dir`. For each flink cluster, there
>> would be a
>> dedicated subdirectory, with the pattern like
>> `${kubernetes.storage.dir}/${cluster-id}`.
>>
>> All resources-related options (e.g. pipeline jars, json plans) that are
>> configured with
>> scheme `file://` <file://`/> <file:///%60 <file:///%60>> would be uploaded to the resource directory
>> and downloaded to the
>> jobmanager, before SQL Driver accesses the files with the original
>> filenames.
>>
>>
>>> * Classloading strategy
>>
>>
>> We could directly specify the SQL Gateway jar as the jar file in
>> PackagedProgram.
>> It would be treated like a normal user jar and the SQL Driver is loaded
>> into the user
>> classloader. WDYT?
>>
>>> * Option `$internal.sql-gateway.driver.sql-config` is string type
>>> I think it's better to use Map type here
>>
>> By Map type configuration, do you mean a nested map that contains all
>> configurations?
>>
>> I hope I've explained myself well, it’s a file that contains the extra SQL
>> configurations, which would be shipped to the jobmanager.
>>
>>> * PoC branch
>>
>> Sure. I’ll let you know once I get the job done.
>>
>> Best,
>> Paul Lam
>>
>>> 2023年6月26日 14:27，Shengkai Fang <fskmine@gmail.com <ma...@gmail.com>> 写道：
>>>
>>> Hi, Paul.
>>>
>>> Thanks for your update. I have a few questions about the new design:
>>>
>>> * How can we ship the json plan to the JobManager?
>>>
>>> The current design only exposes an option about the URL of the json
>> plan. It seems the gateway is responsible to upload to an external stroage.
>> Can we reuse the PipelineOptions.JARS to ship to the remote filesystem?
>>>
>>> * Classloading strategy
>>>
>>> Currently, the Driver is in the sql-gateway package. It means the Driver
>> is not in the JM's classpath directly. Because the sql-gateway jar is now
>> in the opt directory rather than lib directory. It may need to add the
>> external dependencies as Python does[1]. BTW, I think it's better to move
>> the Driver into the flink-table-runtime package, which is much easier to
>> find(Sorry for the wrong opinion before).
>>>
>>> * Option `$internal.sql-gateway.driver.sql-config` is string type
>>>
>>> I think it's better to use Map type here
>>>
>>> * PoC branch
>>>
>>> Because this FLIP involves many modules, do you have a PoC branch to
>> verify it does work?
>>>
>>> Best,
>>> Shengkai
>>>
>>> [1]
>> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 <https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940>
>> <
>> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 <https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940>
>>>
>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>> 于2023年6月19日周一 14:09写道：
>>> Hi Shengkai,
>>>
>>> Sorry for my late reply. It took me some time to update the FLIP.
>>>
>>> In the latest FLIP design, SQL Driver is placed in flink-sql-gateway
>> module. PTAL.
>>>
>>> The FLIP does not cover details about the K8s file distribution, but its
>> general usage would
>>> be very much the same as YARN setups. We could make follow-up
>> discussions in the jira
>>> tickets.
>>>
>>> Best,
>>> Paul Lam
>>>
>>>> 2023年6月12日 15:29，Shengkai Fang <fskmine@gmail.com <ma...@gmail.com> <mailto:
>> fskmine@gmail.com <ma...@gmail.com>>> 写道：
>>>>
>>>>
>>>>> If it’s the case, I’m good with introducing a new module and making
>> SQL Driver
>>>>> an internal class and accepts JSON plans only.
>>>>
>>>> I rethink this again and again. I think it's better to move the
>> SqlDriver into the sql-gateway module because the sql client relies on the
>> sql-gateway to submit the sql and the sql-gateway has the ability to
>> generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
>>>>
>>>> * Upload configuration through command line parameter
>>>>
>>>> ExecNodeGraph only contains the job's information but it doesn't
>> contain the checkpoint dir, checkpoint interval, execution mode and so on.
>> So I think we should also upload the configuration.
>>>>
>>>> * KubernetesClusterDescripter and
>> KubernetesApplicationClusterEntrypoint are responsible for the jar
>> upload/download
>>>>
>>>> +1 for the change.
>>>>
>>>> Could you update the FLIP about the current discussion?
>>>>
>>>> Best,
>>>> Shengkai
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Yang Wang <wangyang0918@apache.org <ma...@apache.org> <mailto:wangyang0918@apache.org <ma...@apache.org>>>
>> 于2023年6月12日周一 11:41写道：
>>>> Sorry for the late reply. I am in favor of introducing such a built-in
>>>> resource localization mechanism
>>>> based on Flink FileSystem. Then FLINK-28915[1] could be the second step
>>>> which will download
>>>> the jars and dependencies to the JobManager/TaskManager local directory
>>>> before working.
>>>>
>>>> The first step could be done in another ticket in Flink. Or some
>> external
>>>> Flink jobs management system
>>>> could also take care of this.
>>>>
>>>> [1]. https://issues.apache.org/jira/browse/FLINK-28915 <https://issues.apache.org/jira/browse/FLINK-28915> <
>> https://issues.apache.org/jira/browse/FLINK-28915 <https://issues.apache.org/jira/browse/FLINK-28915>>
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>> 于2023年6月9日周五 17:39写道：
>>>>
>>>>> Hi Mason,
>>>>>
>>>>> I get your point. I'm increasingly feeling the need to introduce a
>>>>> built-in
>>>>> file distribution mechanism for flink-kubernetes module, just like
>> Spark
>>>>> does with `spark.kubernetes.file.upload.path` [1].
>>>>>
>>>>> I’m assuming the workflow is as follows:
>>>>>
>>>>> - KubernetesClusterDescripter uploads all local resources to a remote
>>>>> storage via Flink filesystem (skips if the resources are already
>> remote).
>>>>> - KubernetesApplicationClusterEntrypoint downloads the resources
>>>>> and put them in the classpath during startup.
>>>>>
>>>>> I wouldn't mind splitting it into another FLIP to ensure that
>> everything is
>>>>> done correctly.
>>>>>
>>>>> cc'ed @Yang to gather more opinions.
>>>>>
>>>>> [1]
>>>>>
>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management <https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management>
>> <
>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management <https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management>
>>>
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>> 2023年6月8日 12:15，Mason Chen <mas.chen6345@gmail.com <ma...@gmail.com> <mailto:
>> mas.chen6345@gmail.com <ma...@gmail.com>>> 写道：
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Thanks for your response!
>>>>>
>>>>> I agree that utilizing SQL Drivers in Java applications is equally
>>>>> important
>>>>>
>>>>> as employing them in SQL Gateway. WRT init containers, I think most
>>>>> users use them just as a workaround. For example, wget a jar from the
>>>>> maven repo.
>>>>>
>>>>> We could implement the functionality in SQL Driver in a more graceful
>>>>> way and the flink-supported filesystem approach seems to be a
>>>>> good choice.
>>>>>
>>>>>
>>>>> My main point is: can we solve the problem with a design agnostic of
>> SQL
>>>>> and Stream API? I mentioned a use case where this ability is useful
>> for
>>>>> Java or Stream API applications. Maybe this is even a non-goal to
>> your FLIP
>>>>> since you are focusing on the driver entrypoint.
>>>>>
>>>>> Jark mentioned some optimizations:
>>>>>
>>>>> This allows SQLGateway to leverage some metadata caching and UDF JAR
>>>>> caching for better compiling performance.
>>>>>
>>>>> It would be great to see this even outside the SQLGateway (i.e. UDF
>> JAR
>>>>> caching).
>>>>>
>>>>> Best,
>>>>> Mason
>>>>>
>>>>> On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskmine@gmail.com <ma...@gmail.com>
>> <mailto:fskmine@gmail.com <ma...@gmail.com>>> wrote:
>>>>>
>>>>> Hi. Paul. Thanks for your update and the update makes me understand
>> the
>>>>> design much better.
>>>>>
>>>>> But I still have some questions about the FLIP.
>>>>>
>>>>> For SQL Gateway, only DMLs need to be delegated to the SQL server
>>>>> Driver. I would think about the details and update the FLIP. Do you
>> have
>>>>>
>>>>> some
>>>>>
>>>>> ideas already?
>>>>>
>>>>>
>>>>> If the applicaiton mode can not support library mode, I think we
>> should
>>>>> only execute INSERT INTO and UPDATE/ DELETE statement in the
>> application
>>>>> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
>>>>> statements. The ANALYZE TABLE syntax need to register the statistic
>> to the
>>>>> catalog after job finishes and the CALL PROCEDURE statement doesn't
>>>>> generate the ExecNodeGraph.
>>>>>
>>>>> * Introduce storage via option `sql-gateway.application.storage-dir`
>>>>>
>>>>> If we can not support to submit the jars through web submission, +1 to
>>>>> introduce the options to upload the files. While I think the uploader
>>>>> should be responsible to remove the uploaded jars. Can we remove the
>> jars
>>>>> if the job is running or gateway exits?
>>>>>
>>>>> * JobID is not avaliable
>>>>>
>>>>> Can we use the returned rest client by ApplicationDeployer to query
>> the job
>>>>> id? I am concerned that users don't know which job is related to the
>>>>> submitted SQL.
>>>>>
>>>>> * Do we need to introduce a new module named flink-table-sql-runner?
>>>>>
>>>>> It seems we need to introduce a new module. Will the new module is
>>>>> available in the distribution package? I agree with Jark that we
>> don't need
>>>>> to introduce this for table-API users and these users have their main
>>>>> class. If we want to make users write the k8s operator more easily, I
>> think
>>>>> we should modify the k8s operator repo. If we don't need to support
>> SQL
>>>>> files, can we make this jar only visible in the sql-gateway like we
>> do in
>>>>> the planner loader?[1]
>>>>>
>>>>> [1]
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 <https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95>
>> <
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 <https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95>
>>>
>>>>>
>>>>> Best,
>>>>> Shengkai
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>>>
>> 于2023年6月7日周三 10:52写道：
>>>>>
>>>>> Hi,
>>>>>
>>>>> Thanks for updating the FLIP.
>>>>>
>>>>> I have two cents on the distribution of SQLs and resources.
>>>>> 1. Should we support a common file distribution mechanism for k8s
>>>>> application mode?
>>>>> I have seen some issues and requirements on the mailing list.
>>>>> In our production environment, we implement the download command in
>> the
>>>>> CliFrontend.
>>>>> And automatically add an init container to the POD for file
>>>>>
>>>>> downloading.
>>>>>
>>>>> The advantage of this
>>>>> is that we can use all Flink-supported file systems to store files.
>>>>>
>>>>> This need more discussion. I would appreciate hearing more opinions.
>>>>>
>>>>> 2. In this FLIP, we distribute files in two different ways in YARN and
>>>>> Kubernetes. Can we combine it in one way?
>>>>> If we don't want to implement a common file distribution for k8s
>>>>> application mode. Could we use the SQLDriver
>>>>> to download the files both in YARN and K8S? IMO, this can reduce the
>>>>>
>>>>> cost
>>>>>
>>>>> of code maintenance.
>>>>>
>>>>> Best,
>>>>> Weihua
>>>>>
>>>>>
>>>>> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>>>>>
>>>>> Hi Mason,
>>>>>
>>>>> Thanks for your input!
>>>>>
>>>>> +1 for init containers or a more generalized way of obtaining
>>>>>
>>>>> arbitrary
>>>>>
>>>>> files. File fetching isn't specific to just SQL--it also matters for
>>>>>
>>>>> Java
>>>>>
>>>>> applications if the user doesn't want to rebuild a Flink image and
>>>>>
>>>>> just
>>>>>
>>>>> wants to modify the user application fat jar.
>>>>>
>>>>>
>>>>> I agree that utilizing SQL Drivers in Java applications is equally
>>>>> important
>>>>> as employing them in SQL Gateway. WRT init containers, I think most
>>>>> users use them just as a workaround. For example, wget a jar from the
>>>>> maven repo.
>>>>>
>>>>> We could implement the functionality in SQL Driver in a more graceful
>>>>> way and the flink-supported filesystem approach seems to be a
>>>>> good choice.
>>>>>
>>>>> Also, what do you think about prefixing the config options with
>>>>> `sql-driver` instead of just `sql` to be more specific?
>>>>>
>>>>>
>>>>> LGTM, since SQL Driver is a public interface and the options are
>>>>> specific to it.
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>> 2023年6月6日 06:30，Mason Chen <mas.chen6345@gmail.com <ma...@gmail.com> <mailto:
>> mas.chen6345@gmail.com <ma...@gmail.com>>> 写道：
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> +1 for this feature and supporting SQL file + JSON plans. We get a
>>>>>
>>>>> lot
>>>>>
>>>>> of
>>>>>
>>>>> requests to just be able to submit a SQL file, but the JSON plan
>>>>> optimizations make sense.
>>>>>
>>>>> +1 for init containers or a more generalized way of obtaining
>>>>>
>>>>> arbitrary
>>>>>
>>>>> files. File fetching isn't specific to just SQL--it also matters for
>>>>>
>>>>> Java
>>>>>
>>>>> applications if the user doesn't want to rebuild a Flink image and
>>>>>
>>>>> just
>>>>>
>>>>> wants to modify the user application fat jar.
>>>>>
>>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>>>>
>>>>> which
>>>>>
>>>>> should
>>>>>
>>>>>
>>>>> be required to run Flink in production, so I guess that would be
>>>>>
>>>>> acceptable
>>>>>
>>>>> for most
>>>>>
>>>>>
>>>>> users. WDYT?
>>>>>
>>>>>
>>>>> If you do go this route, it would be nice to support writing these
>>>>>
>>>>> files
>>>>>
>>>>> to
>>>>>
>>>>> S3/HDFS via Flink. This makes access control and policy management
>>>>>
>>>>> simpler.
>>>>>
>>>>>
>>>>> Also, what do you think about prefixing the config options with
>>>>> `sql-driver` instead of just `sql` to be more specific?
>>>>>
>>>>> Best,
>>>>> Mason
>>>>>
>>>>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>
>>>>>
>>>>> <mailto:
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>>>>>
>>>>>
>>>>> Hi Jark,
>>>>>
>>>>> Thanks for your input! Please see my comments inline.
>>>>>
>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>> DataStream API also doesn't provide a default main class for users,
>>>>> why do we need to provide such one for SQL?
>>>>>
>>>>>
>>>>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>>>>>
>>>>> submitted
>>>>>
>>>>> via Flink CLI which actually could be DataStream/Table jobs.
>>>>>
>>>>> I think a default main class would be user-friendly which eliminates
>>>>>
>>>>> the
>>>>>
>>>>> need
>>>>> for users to write a main class as SQLRunner in Flink K8s operator
>>>>>
>>>>> [1].
>>>>>
>>>>>
>>>>> I thought the proposed SqlDriver was a dedicated main class
>>>>>
>>>>> accepting
>>>>>
>>>>> SQL files, is
>>>>>
>>>>> that correct?
>>>>>
>>>>>
>>>>> Both JSON plans and SQL files are accepted. SQL Gateway should use
>>>>>
>>>>> JSON
>>>>>
>>>>> plans,
>>>>> while CLI users may use either JSON plans or SQL files.
>>>>>
>>>>> Please see the updated FLIP[2] for more details.
>>>>>
>>>>> Personally, I prefer the way of init containers which doesn't
>>>>>
>>>>> depend
>>>>>
>>>>> on
>>>>>
>>>>> additional components.
>>>>> This can reduce the moving parts of a production environment.
>>>>> Depending on a distributed file system makes the testing, demo, and
>>>>>
>>>>> local
>>>>>
>>>>> setup harder than init containers.
>>>>>
>>>>>
>>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>>>>
>>>>> which
>>>>>
>>>>> should
>>>>> be required to run Flink in production, so I guess that would be
>>>>> acceptable for most
>>>>> users. WDYT?
>>>>>
>>>>> WRT testing, demo, and local setups, I think we could support the
>>>>>
>>>>> local
>>>>>
>>>>> filesystem
>>>>> scheme i.e. file://** <file:///**> <> as the state backends do. It works as long as
>>>>>
>>>>> SQL
>>>>>
>>>>> Gateway
>>>>> and JobManager(or SQL Driver) can access the resource directory
>>>>>
>>>>> (specified
>>>>>
>>>>> via
>>>>> `sql-gateway.application.storage-dir`).
>>>>>
>>>>> Thanks!
>>>>>
>>>>> [1]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java <https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java>
>> <
>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java <https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java>
>>>
>>>>>
>>>>> [2]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
>> <
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
>>>
>>>>>
>>>>> [3]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 <https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161>
>> <
>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 <https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161>
>>>
>>>>>
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>> 2023年6月3日 12:21，Jark Wu <imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>
>> 写道：
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Thanks for your reply. I left my comments inline.
>>>>>
>>>>> As the FLIP said, it’s good to have a default main class for Flink
>>>>>
>>>>> SQLs,
>>>>>
>>>>> which allows users to submit Flink SQLs in the same way as
>>>>>
>>>>> DataStream
>>>>>
>>>>> jobs, or else users need to write their own main class.
>>>>>
>>>>>
>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>> DataStream API also doesn't provide a default main class for users,
>>>>> why do we need to provide such one for SQL?
>>>>>
>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>> SessionState? If not, we could make SQL Driver accepts two
>>>>>
>>>>> serialized
>>>>>
>>>>> formats:
>>>>>
>>>>>
>>>>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
>>>>>
>>>>> the
>>>>>
>>>>> proposed SqlDriver was a dedicated main class accepting SQL files,
>>>>>
>>>>> is
>>>>>
>>>>> that correct?
>>>>> If true, we have to ship the SessionState for this case which is a
>>>>>
>>>>> large
>>>>>
>>>>> work.
>>>>> I think we just need a JsonPlanDriver which is a main class that
>>>>>
>>>>> accepts
>>>>>
>>>>> JsonPlan as the parameter.
>>>>>
>>>>>
>>>>> The common solutions I know is to use distributed file systems or
>>>>>
>>>>> use
>>>>>
>>>>> init containers to localize the resources.
>>>>>
>>>>>
>>>>> Personally, I prefer the way of init containers which doesn't
>>>>>
>>>>> depend
>>>>>
>>>>> on
>>>>>
>>>>> additional components.
>>>>> This can reduce the moving parts of a production environment.
>>>>> Depending on a distributed file system makes the testing, demo, and
>>>>>
>>>>> local
>>>>>
>>>>> setup harder than init containers.
>>>>>
>>>>> Best,
>>>>> Jark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>>
>>>>>
>>>>> <mailto:
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>> <mailto:
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>> wrote:
>>>>>
>>>>>
>>>>> The FLIP is in the early phase and some details are not included,
>>>>>
>>>>> but
>>>>>
>>>>> fortunately, we got lots of valuable ideas from the discussion.
>>>>>
>>>>> Thanks to everyone who joined the dissuasion!
>>>>> @Weihua @Shanmon @Shengkai @Biao @Jark
>>>>>
>>>>> This weekend I’m gonna revisit and update the FLIP, adding more
>>>>> details. Hopefully, we can further align our opinions.
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>> 写道：
>>>>>
>>>>>
>>>>> Hi Jark,
>>>>>
>>>>> Thanks a lot for your input!
>>>>>
>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>>>>
>>>>> still
>>>>>
>>>>> necessary to support SQL Driver?
>>>>>
>>>>>
>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
>>>>>
>>>>> simplify
>>>>>
>>>>> Flink SQL execution with Flink CLI.
>>>>>
>>>>> As the FLIP said, it’s good to have a default main class for
>>>>>
>>>>> Flink
>>>>>
>>>>> SQLs,
>>>>>
>>>>> which allows users to submit Flink SQLs in the same way as
>>>>>
>>>>> DataStream
>>>>>
>>>>> jobs, or else users need to write their own main class.
>>>>>
>>>>> SQL Driver needs to serialize SessionState which is very
>>>>>
>>>>> challenging
>>>>>
>>>>> but not detailed covered in the FLIP.
>>>>>
>>>>>
>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>> SessionState? If not, we could make SQL Driver accepts two
>>>>>
>>>>> serialized
>>>>>
>>>>> formats:
>>>>>
>>>>> - SQL files for user-facing public usage
>>>>> - ExecNodeGraph for internal usage
>>>>>
>>>>> It’s kind of similar to the relationship between job jars and
>>>>>
>>>>> jobgraphs.
>>>>>
>>>>>
>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>>>>>
>>>>> true?
>>>>>
>>>>> Is it
>>>>>
>>>>> possible to support it?
>>>>>
>>>>>
>>>>> Yes, K8s doesn’t distribute any files. It’s the users’
>>>>>
>>>>> responsibility
>>>>>
>>>>> to
>>>>>
>>>>> make
>>>>>
>>>>> sure the resources are accessible in the containers. The common
>>>>>
>>>>> solutions
>>>>>
>>>>> I know is to use distributed file systems or use init containers
>>>>>
>>>>> to
>>>>>
>>>>> localize the
>>>>>
>>>>> resources.
>>>>>
>>>>> Now I lean toward introducing a fs to do the distribution job.
>>>>>
>>>>> WDYT?
>>>>>
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>
>> <mailto:
>>>>>
>>>>> imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>
>>>>>
>>>>> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>> <mailto:
>> imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>>
>>>>>
>>>>> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>> <mailto:
>> imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>> <mailto:
>>>>>
>>>>> imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>> <mailto:imjark@gmail.com <ma...@gmail.com>
>> <mailto:imjark@gmail.com <ma...@gmail.com>>>>>>
>>>>>
>>>>> 写道：
>>>>>
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Thanks for starting this discussion. I like the proposal! This
>>>>>
>>>>> is
>>>>>
>>>>> a
>>>>>
>>>>> frequently requested feature!
>>>>>
>>>>> I agree with Shengkai that ExecNodeGraph as the submission
>>>>>
>>>>> object
>>>>>
>>>>> is a
>>>>>
>>>>> better idea than SQL file. To be more specific, it should be
>>>>>
>>>>> JsonPlanGraph
>>>>>
>>>>> or CompiledPlan which is the serializable representation.
>>>>>
>>>>> CompiledPlan
>>>>>
>>>>> is a
>>>>>
>>>>> clear separation between compiling/optimization/validation and
>>>>>
>>>>> execution.
>>>>>
>>>>> This can keep the validation and metadata accessing still on the
>>>>>
>>>>> SQLGateway
>>>>>
>>>>> side. This allows SQLGateway to leverage some metadata caching
>>>>>
>>>>> and
>>>>>
>>>>> UDF
>>>>>
>>>>> JAR
>>>>>
>>>>> caching for better compiling performance.
>>>>>
>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>>>>
>>>>> still
>>>>>
>>>>> necessary to support SQL Driver? Regarding non-interactive SQL
>>>>>
>>>>> jobs,
>>>>>
>>>>> users
>>>>>
>>>>> can use the Table API program for application mode. SQL Driver
>>>>>
>>>>> needs
>>>>>
>>>>> to
>>>>>
>>>>> serialize SessionState which is very challenging but not
>>>>>
>>>>> detailed
>>>>>
>>>>> covered
>>>>>
>>>>> in the FLIP.
>>>>>
>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>>>>>
>>>>> true?
>>>>>
>>>>> Is it
>>>>>
>>>>> possible to support it?
>>>>>
>>>>> Best,
>>>>> Jark
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>>
>>>>>
>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>> <mailto:
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>> <mailto:
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>> <mailto:
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>>> wrote:
>>>>>
>>>>>
>>>>> Hi Weihua,
>>>>>
>>>>> You’re right. Distributing the SQLs to the TMs is one of the
>>>>>
>>>>> challenging
>>>>>
>>>>> parts of this FLIP.
>>>>>
>>>>> Web submission is not enabled in application mode currently as
>>>>>
>>>>> you
>>>>>
>>>>> said,
>>>>>
>>>>> but it could be changed if we have good reasons.
>>>>>
>>>>> What do you think about introducing a distributed storage for
>>>>>
>>>>> SQL
>>>>>
>>>>> Gateway?
>>>>>
>>>>>
>>>>> We could make use of Flink file systems [1] to distribute the
>>>>>
>>>>> SQL
>>>>>
>>>>> Gateway
>>>>>
>>>>> generated resources, that should solve the problem at its root
>>>>>
>>>>> cause.
>>>>>
>>>>>
>>>>> Users could specify Flink-supported file systems to ship files.
>>>>>
>>>>> It’s
>>>>>
>>>>> only
>>>>>
>>>>> required when using SQL Gateway with K8s application mode.
>>>>>
>>>>> [1]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
>> huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
>>>>>
>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>>> <mailto:
>>>>>
>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
>> huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>>>>> 写道：
>>>>>
>>>>>
>>>>> Thanks Paul for your reply.
>>>>>
>>>>> SQLDriver looks good to me.
>>>>>
>>>>> 2. Do you mean a pass the SQL string a configuration or a
>>>>>
>>>>> program
>>>>>
>>>>> argument?
>>>>>
>>>>>
>>>>>
>>>>> I brought this up because we were unable to pass the SQL file
>>>>>
>>>>> to
>>>>>
>>>>> Flink
>>>>>
>>>>> using Kubernetes mode.
>>>>> For DataStream/Python users, they need to prepare their images
>>>>>
>>>>> for
>>>>>
>>>>> the
>>>>>
>>>>> jars
>>>>>
>>>>> and dependencies.
>>>>> But for SQL users, they can use a common image to run
>>>>>
>>>>> different
>>>>>
>>>>> SQL
>>>>>
>>>>> queries
>>>>>
>>>>> if there are no other udf requirements.
>>>>> It would be great if the SQL query and image were not bound.
>>>>>
>>>>> Using strings is a way to decouple these, but just as you
>>>>>
>>>>> mentioned,
>>>>>
>>>>> it's
>>>>>
>>>>> not easy to pass complex SQL.
>>>>>
>>>>> use web submission
>>>>>
>>>>> AFAIK, we can not use web submission in the Application mode.
>>>>>
>>>>> Please
>>>>>
>>>>> correct me if I'm wrong.
>>>>>
>>>>>
>>>>> Best,
>>>>> Weihua
>>>>>
>>>>>
>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>
>>>>>
>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>>>
>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>>
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hi Biao,
>>>>>
>>>>> Thanks for your comments!
>>>>>
>>>>> 1. Scope: is this FLIP only targeted for non-interactive
>>>>>
>>>>> Flink
>>>>>
>>>>> SQL
>>>>>
>>>>> jobs
>>>>>
>>>>> in
>>>>>
>>>>> Application mode? More specifically, if we use SQL
>>>>>
>>>>> client/gateway
>>>>>
>>>>> to
>>>>>
>>>>> execute some interactive SQLs like a SELECT query, can we
>>>>>
>>>>> ask
>>>>>
>>>>> flink
>>>>>
>>>>> to
>>>>>
>>>>> use
>>>>>
>>>>> Application mode to execute those queries after this FLIP?
>>>>>
>>>>>
>>>>> Thanks for pointing it out. I think only DMLs would be
>>>>>
>>>>> executed
>>>>>
>>>>> via
>>>>>
>>>>> SQL
>>>>>
>>>>> Driver.
>>>>> I'll add the scope to the FLIP.
>>>>>
>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>
>>>>> trivial as
>>>>>
>>>>> we
>>>>>
>>>>> can ship files via YARN's tool easily but for K8s, things
>>>>>
>>>>> can
>>>>>
>>>>> be
>>>>>
>>>>> more
>>>>>
>>>>> complicated as Shengkai said.
>>>>>
>>>>>
>>>>>
>>>>> Your input is very informative. I’m thinking about using web
>>>>>
>>>>> submission,
>>>>>
>>>>> but it requires exposing the JobManager port which could also
>>>>>
>>>>> be
>>>>>
>>>>> a
>>>>>
>>>>> problem
>>>>>
>>>>> on K8s.
>>>>>
>>>>> Another approach is to explicitly require a distributed
>>>>>
>>>>> storage
>>>>>
>>>>> to
>>>>>
>>>>> ship
>>>>>
>>>>> files,
>>>>> but we may need a new deployment executor for that.
>>>>>
>>>>> What do you think of these two approaches?
>>>>>
>>>>> 3. Serialization of SessionState: in SessionState, there are
>>>>>
>>>>> some
>>>>>
>>>>> unserializable fields
>>>>> like
>>>>>
>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>
>>>>> It
>>>>>
>>>>> may be worthwhile to add more details about the
>>>>>
>>>>> serialization
>>>>>
>>>>> part.
>>>>>
>>>>>
>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph
>>>>>
>>>>> as
>>>>>
>>>>> Shengkai
>>>>>
>>>>> mentioned, do we eliminate the need for serialization of
>>>>>
>>>>> SessionState?
>>>>>
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <ma...@gmail.com> <mailto:
>> biaogeng7@gmail.com <ma...@gmail.com>> <mailto:
>>>>>
>>>>> biaogeng7@gmail.com <ma...@gmail.com> <mailto:biaogeng7@gmail.com <ma...@gmail.com>>> <mailto:
>>>>>
>>>>> biaogeng7@gmail.com <ma...@gmail.com> <mailto:biaogeng7@gmail.com <ma...@gmail.com>> <mailto:
>> biaogeng7@gmail.com <ma...@gmail.com> <mailto:biaogeng7@gmail.com <ma...@gmail.com>>>>> 写道：
>>>>>
>>>>>
>>>>> Thanks Paul for the proposal!I believe it would be very
>>>>>
>>>>> useful
>>>>>
>>>>> for
>>>>>
>>>>> flink
>>>>>
>>>>> users.
>>>>> After reading the FLIP, I have some questions:
>>>>> 1. Scope: is this FLIP only targeted for non-interactive
>>>>>
>>>>> Flink
>>>>>
>>>>> SQL
>>>>>
>>>>> jobs
>>>>>
>>>>> in
>>>>>
>>>>> Application mode? More specifically, if we use SQL
>>>>>
>>>>> client/gateway
>>>>>
>>>>> to
>>>>>
>>>>> execute some interactive SQLs like a SELECT query, can we
>>>>>
>>>>> ask
>>>>>
>>>>> flink
>>>>>
>>>>> to
>>>>>
>>>>> use
>>>>>
>>>>> Application mode to execute those queries after this FLIP?
>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>
>>>>> trivial as
>>>>>
>>>>> we
>>>>>
>>>>> can ship files via YARN's tool easily but for K8s, things
>>>>>
>>>>> can
>>>>>
>>>>> be
>>>>>
>>>>> more
>>>>>
>>>>> complicated as Shengkai said. I have implemented a simple
>>>>>
>>>>> POC
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> <
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> <
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>>>
>>>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> <
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> <
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> based on SQL client before(i.e. consider the SQL client
>>>>>
>>>>> which
>>>>>
>>>>> supports
>>>>>
>>>>> executing a SQL file as the SQL driver in this FLIP). One
>>>>>
>>>>> problem
>>>>>
>>>>> I
>>>>>
>>>>> have
>>>>>
>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s
>>>>>
>>>>> side.
>>>>>
>>>>> Without
>>>>>
>>>>> such support, users have to modify the initContainer or
>>>>>
>>>>> rebuild
>>>>>
>>>>> a
>>>>>
>>>>> new
>>>>>
>>>>> K8s
>>>>>
>>>>> image every time to fetch the SQL file. Like the flink k8s
>>>>>
>>>>> operator,
>>>>>
>>>>> one
>>>>>
>>>>> workaround is to utilize the flink config(transforming the
>>>>>
>>>>> SQL
>>>>>
>>>>> file
>>>>>
>>>>> to
>>>>>
>>>>> a
>>>>>
>>>>> escaped string like Weihua mentioned) which will be
>>>>>
>>>>> converted
>>>>>
>>>>> to a
>>>>>
>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger
>>>>>
>>>>> than
>>>>>
>>>>> 1MB
>>>>>
>>>>> <
>>>>>
>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>>
>>>>>
>>>>> <
>>>>>
>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>>> <
>>>>>
>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>> <
>>>>>
>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>>>>>).
>>>>>
>>>>> Not
>>>>>
>>>>> sure
>>>>>
>>>>> if we have better solutions.
>>>>> 3. Serialization of SessionState: in SessionState, there are
>>>>>
>>>>> some
>>>>>
>>>>> unserializable fields
>>>>> like
>>>>>
>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>
>>>>> It
>>>>>
>>>>> may be worthwhile to add more details about the
>>>>>
>>>>> serialization
>>>>>
>>>>> part.
>>>>>
>>>>>
>>>>> Best,
>>>>> Biao Geng
>>>>>
>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>
>> <mailto:
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>
>>>>>
>>>>>
>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>>
>>>>>
>>>>> 于2023年5月31日周三 11:49写道：
>>>>>
>>>>>
>>>>> Hi Weihua,
>>>>>
>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>
>>>>> - Is SQLRunner the better name? We use this to run a SQL
>>>>>
>>>>> Job.
>>>>>
>>>>> (Not
>>>>>
>>>>> strong,
>>>>>
>>>>> the SQLDriver is fine for me)
>>>>>
>>>>>
>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
>>>>>
>>>>> following
>>>>>
>>>>> reasons FYI:
>>>>>
>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>> 2. Flink program's main class is sort of like Driver in
>>>>>
>>>>> JDBC
>>>>>
>>>>> which
>>>>>
>>>>> translates SQLs into
>>>>> databases specific languages.
>>>>>
>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>
>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
>>>>>
>>>>> we
>>>>>
>>>>> need
>>>>>
>>>>> to
>>>>>
>>>>> prepare
>>>>>
>>>>> a SQL file in an image for Kubernetes application mode,
>>>>>
>>>>> which
>>>>>
>>>>> may
>>>>>
>>>>> be
>>>>>
>>>>> a
>>>>>
>>>>> bit
>>>>>
>>>>> cumbersome.
>>>>>
>>>>>
>>>>> Do you mean a pass the SQL string a configuration or a
>>>>>
>>>>> program
>>>>>
>>>>> argument?
>>>>>
>>>>>
>>>>> I thought it might be convenient for testing propose, but
>>>>>
>>>>> not
>>>>>
>>>>> recommended
>>>>>
>>>>> for production,
>>>>> cause Flink SQLs could be complicated and involves lots of
>>>>>
>>>>> characters
>>>>>
>>>>> that
>>>>>
>>>>> need to escape.
>>>>>
>>>>> WDYT?
>>>>>
>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>
>>>>> "run-application"
>>>>>
>>>>> command. Does that mean we need to perform automatic
>>>>>
>>>>> detection
>>>>>
>>>>> in
>>>>>
>>>>> Flink?
>>>>>
>>>>>
>>>>> Yes! It’s like running a PyFlink job with the following
>>>>>
>>>>> command:
>>>>>
>>>>>
>>>>> ```
>>>>> ./bin/flink run \
>>>>> --pyModule table.word_count \
>>>>> --pyFiles examples/python/table
>>>>> ```
>>>>>
>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
>>>>>
>>>>> Driver
>>>>>
>>>>> automatically.
>>>>>
>>>>>
>>>>> [1]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> <
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> <
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>>>
>>>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> <
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> <
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
>> huweihua.ckl@gmail.com <ma...@gmail.com>>
>>>>>
>>>>> <mailto:
>>>>>
>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>>> <mailto:
>>>>>
>>>>> huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>>>> 写道：
>>>>>
>>>>>
>>>>> Thanks Paul for the proposal.
>>>>>
>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>
>>>>> I have a few questions.
>>>>> - Is SQLRunner the better name? We use this to run a SQL
>>>>>
>>>>> Job.
>>>>>
>>>>> (Not
>>>>>
>>>>> strong,
>>>>>
>>>>> the SQLDriver is fine for me)
>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
>>>>>
>>>>> we
>>>>>
>>>>> need
>>>>>
>>>>> to
>>>>>
>>>>> prepare
>>>>>
>>>>> a SQL file in an image for Kubernetes application mode,
>>>>>
>>>>> which
>>>>>
>>>>> may
>>>>>
>>>>> be
>>>>>
>>>>> a
>>>>>
>>>>> bit
>>>>>
>>>>> cumbersome.
>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>
>>>>> "run-application"
>>>>>
>>>>> command. Does that mean we need to perform automatic
>>>>>
>>>>> detection
>>>>>
>>>>> in
>>>>>
>>>>> Flink?
>>>>>
>>>>>
>>>>>
>>>>> Best,
>>>>> Weihua
>>>>>
>>>>>
>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>>>>>
>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>
>>>>>
>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>>
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hi team,
>>>>>
>>>>> I’d like to start a discussion about FLIP-316 [1], which
>>>>>
>>>>> introduces
>>>>>
>>>>> a
>>>>>
>>>>> SQL
>>>>>
>>>>> driver as the
>>>>> default main class for Flink SQL jobs.
>>>>>
>>>>> Currently, Flink SQL could be executed out of the box
>>>>>
>>>>> either
>>>>>
>>>>> via
>>>>>
>>>>> SQL
>>>>>
>>>>> Client/Gateway
>>>>> or embedded in a Flink Java/Python program.
>>>>>
>>>>> However, each one has its drawback:
>>>>>
>>>>> - SQL Client/Gateway doesn’t support the application
>>>>>
>>>>> deployment
>>>>>
>>>>> mode
>>>>>
>>>>> [2]
>>>>>
>>>>> - Flink Java/Python program requires extra work to write
>>>>>
>>>>> a
>>>>>
>>>>> non-SQL
>>>>>
>>>>> program
>>>>>
>>>>>
>>>>> Therefore, I propose adding a SQL driver to act as the
>>>>>
>>>>> default
>>>>>
>>>>> main
>>>>>
>>>>> class
>>>>>
>>>>> for SQL jobs.
>>>>> Please see the FLIP docs for details and feel free to
>>>>>
>>>>> comment.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> [1]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver>
>> <
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver>
>>>
>>>>>
>>>>> <
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
>> <
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
>>>
>>>>>
>>>>>
>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541> <
>> https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541>> <
>>>>> https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541> <
>> https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541>>>
>>>>>
>>>>> Best,
>>>>> Paul Lam
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Paul,
Thanks for driving it and thank you all for the informative discussion! The
FLIP is in good shape now. As described in the FLIP, SQL Driver will be
mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway in
application mode and 2. external system integration. Would you like to add
one section to describe(better with script/code example) how to use it in
these two scenarios from users' perspective?
NIT: the pictures have transparent background when readers click on it. It
would be great if you can replace them with pictures with white background.
Best regards,
Jing
On Mon, Jun 26, 2023 at 1:31 PM Paul Lam <pa...@gmail.com> wrote:
> Hi Shengkai,
>
> > * How can we ship the json plan to the JobManager?
>
> The Flink K8s module should be responsible for file distribution. We could
> introduce
> an option like `kubernetes.storage.dir`. For each flink cluster, there
> would be a
> dedicated subdirectory, with the pattern like
> `${kubernetes.storage.dir}/${cluster-id}`.
>
> All resources-related options (e.g. pipeline jars, json plans) that are
> configured with
> scheme `file://` <file:///%60> would be uploaded to the resource directory
> and downloaded to the
> jobmanager, before SQL Driver accesses the files with the original
> filenames.
>
>
> > * Classloading strategy
>
>
> We could directly specify the SQL Gateway jar as the jar file in
> PackagedProgram.
> It would be treated like a normal user jar and the SQL Driver is loaded
> into the user
> classloader. WDYT?
>
> > * Option `$internal.sql-gateway.driver.sql-config` is string type
> > I think it's better to use Map type here
>
> By Map type configuration, do you mean a nested map that contains all
> configurations?
>
> I hope I've explained myself well, it’s a file that contains the extra SQL
> configurations, which would be shipped to the jobmanager.
>
> > * PoC branch
>
> Sure. I’ll let you know once I get the job done.
>
> Best,
> Paul Lam
>
> > 2023年6月26日 14:27，Shengkai Fang <fs...@gmail.com> 写道：
> >
> > Hi, Paul.
> >
> > Thanks for your update. I have a few questions about the new design:
> >
> > * How can we ship the json plan to the JobManager?
> >
> > The current design only exposes an option about the URL of the json
> plan. It seems the gateway is responsible to upload to an external stroage.
> Can we reuse the PipelineOptions.JARS to ship to the remote filesystem?
> >
> > * Classloading strategy
> >
> > Currently, the Driver is in the sql-gateway package. It means the Driver
> is not in the JM's classpath directly. Because the sql-gateway jar is now
> in the opt directory rather than lib directory. It may need to add the
> external dependencies as Python does[1]. BTW, I think it's better to move
> the Driver into the flink-table-runtime package, which is much easier to
> find(Sorry for the wrong opinion before).
> >
> > * Option `$internal.sql-gateway.driver.sql-config` is string type
> >
> > I think it's better to use Map type here
> >
> > * PoC branch
> >
> > Because this FLIP involves many modules, do you have a PoC branch to
> verify it does work?
> >
> > Best,
> > Shengkai
> >
> > [1]
> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
> <
> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
> >
> > Paul Lam <paullin3280@gmail.com <ma...@gmail.com>>
> 于2023年6月19日周一 14:09写道：
> > Hi Shengkai,
> >
> > Sorry for my late reply. It took me some time to update the FLIP.
> >
> > In the latest FLIP design, SQL Driver is placed in flink-sql-gateway
> module. PTAL.
> >
> > The FLIP does not cover details about the K8s file distribution, but its
> general usage would
> > be very much the same as YARN setups. We could make follow-up
> discussions in the jira
> > tickets.
> >
> > Best,
> > Paul Lam
> >
> >> 2023年6月12日 15:29，Shengkai Fang <fskmine@gmail.com <mailto:
> fskmine@gmail.com>> 写道：
> >>
> >>
> >> > If it’s the case, I’m good with introducing a new module and making
> SQL Driver
> >> > an internal class and accepts JSON plans only.
> >>
> >> I rethink this again and again. I think it's better to move the
> SqlDriver into the sql-gateway module because the sql client relies on the
> sql-gateway to submit the sql and the sql-gateway has the ability to
> generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
> >>
> >> * Upload configuration through command line parameter
> >>
> >> ExecNodeGraph only contains the job's information but it doesn't
> contain the checkpoint dir, checkpoint interval, execution mode and so on.
> So I think we should also upload the configuration.
> >>
> >> * KubernetesClusterDescripter and
> KubernetesApplicationClusterEntrypoint are responsible for the jar
> upload/download
> >>
> >> +1 for the change.
> >>
> >> Could you update the FLIP about the current discussion?
> >>
> >> Best,
> >> Shengkai
> >>
> >>
> >>
> >>
> >>
> >>
> >> Yang Wang <wangyang0918@apache.org <ma...@apache.org>>
> 于2023年6月12日周一 11:41写道：
> >> Sorry for the late reply. I am in favor of introducing such a built-in
> >> resource localization mechanism
> >> based on Flink FileSystem. Then FLINK-28915[1] could be the second step
> >> which will download
> >> the jars and dependencies to the JobManager/TaskManager local directory
> >> before working.
> >>
> >> The first step could be done in another ticket in Flink. Or some
> external
> >> Flink jobs management system
> >> could also take care of this.
> >>
> >> [1]. https://issues.apache.org/jira/browse/FLINK-28915 <
> https://issues.apache.org/jira/browse/FLINK-28915>
> >>
> >> Best,
> >> Yang
> >>
> >> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>>
> 于2023年6月9日周五 17:39写道：
> >>
> >> > Hi Mason,
> >> >
> >> > I get your point. I'm increasingly feeling the need to introduce a
> >> > built-in
> >> > file distribution mechanism for flink-kubernetes module, just like
> Spark
> >> > does with `spark.kubernetes.file.upload.path` [1].
> >> >
> >> > I’m assuming the workflow is as follows:
> >> >
> >> > - KubernetesClusterDescripter uploads all local resources to a remote
> >> > storage via Flink filesystem (skips if the resources are already
> remote).
> >> > - KubernetesApplicationClusterEntrypoint downloads the resources
> >> > and put them in the classpath during startup.
> >> >
> >> > I wouldn't mind splitting it into another FLIP to ensure that
> everything is
> >> > done correctly.
> >> >
> >> > cc'ed @Yang to gather more opinions.
> >> >
> >> > [1]
> >> >
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> <
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> >
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> > 2023年6月8日 12:15，Mason Chen <mas.chen6345@gmail.com <mailto:
> mas.chen6345@gmail.com>> 写道：
> >> >
> >> > Hi Paul,
> >> >
> >> > Thanks for your response!
> >> >
> >> > I agree that utilizing SQL Drivers in Java applications is equally
> >> > important
> >> >
> >> > as employing them in SQL Gateway. WRT init containers, I think most
> >> > users use them just as a workaround. For example, wget a jar from the
> >> > maven repo.
> >> >
> >> > We could implement the functionality in SQL Driver in a more graceful
> >> > way and the flink-supported filesystem approach seems to be a
> >> > good choice.
> >> >
> >> >
> >> > My main point is: can we solve the problem with a design agnostic of
> SQL
> >> > and Stream API? I mentioned a use case where this ability is useful
> for
> >> > Java or Stream API applications. Maybe this is even a non-goal to
> your FLIP
> >> > since you are focusing on the driver entrypoint.
> >> >
> >> > Jark mentioned some optimizations:
> >> >
> >> > This allows SQLGateway to leverage some metadata caching and UDF JAR
> >> > caching for better compiling performance.
> >> >
> >> > It would be great to see this even outside the SQLGateway (i.e. UDF
> JAR
> >> > caching).
> >> >
> >> > Best,
> >> > Mason
> >> >
> >> > On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskmine@gmail.com
> <ma...@gmail.com>> wrote:
> >> >
> >> > Hi. Paul. Thanks for your update and the update makes me understand
> the
> >> > design much better.
> >> >
> >> > But I still have some questions about the FLIP.
> >> >
> >> > For SQL Gateway, only DMLs need to be delegated to the SQL server
> >> > Driver. I would think about the details and update the FLIP. Do you
> have
> >> >
> >> > some
> >> >
> >> > ideas already?
> >> >
> >> >
> >> > If the applicaiton mode can not support library mode, I think we
> should
> >> > only execute INSERT INTO and UPDATE/ DELETE statement in the
> application
> >> > mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> >> > statements. The ANALYZE TABLE syntax need to register the statistic
> to the
> >> > catalog after job finishes and the CALL PROCEDURE statement doesn't
> >> > generate the ExecNodeGraph.
> >> >
> >> > * Introduce storage via option `sql-gateway.application.storage-dir`
> >> >
> >> > If we can not support to submit the jars through web submission, +1 to
> >> > introduce the options to upload the files. While I think the uploader
> >> > should be responsible to remove the uploaded jars. Can we remove the
> jars
> >> > if the job is running or gateway exits?
> >> >
> >> > * JobID is not avaliable
> >> >
> >> > Can we use the returned rest client by ApplicationDeployer to query
> the job
> >> > id? I am concerned that users don't know which job is related to the
> >> > submitted SQL.
> >> >
> >> > * Do we need to introduce a new module named flink-table-sql-runner?
> >> >
> >> > It seems we need to introduce a new module. Will the new module is
> >> > available in the distribution package? I agree with Jark that we
> don't need
> >> > to introduce this for table-API users and these users have their main
> >> > class. If we want to make users write the k8s operator more easily, I
> think
> >> > we should modify the k8s operator repo. If we don't need to support
> SQL
> >> > files, can we make this jar only visible in the sql-gateway like we
> do in
> >> > the planner loader?[1]
> >> >
> >> > [1]
> >> >
> >> >
> >> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> <
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> >
> >> >
> >> > Best,
> >> > Shengkai
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com>>
> 于2023年6月7日周三 10:52写道：
> >> >
> >> > Hi,
> >> >
> >> > Thanks for updating the FLIP.
> >> >
> >> > I have two cents on the distribution of SQLs and resources.
> >> > 1. Should we support a common file distribution mechanism for k8s
> >> > application mode?
> >> > I have seen some issues and requirements on the mailing list.
> >> > In our production environment, we implement the download command in
> the
> >> > CliFrontend.
> >> > And automatically add an init container to the POD for file
> >> >
> >> > downloading.
> >> >
> >> > The advantage of this
> >> > is that we can use all Flink-supported file systems to store files.
> >> >
> >> > This need more discussion. I would appreciate hearing more opinions.
> >> >
> >> > 2. In this FLIP, we distribute files in two different ways in YARN and
> >> > Kubernetes. Can we combine it in one way?
> >> > If we don't want to implement a common file distribution for k8s
> >> > application mode. Could we use the SQLDriver
> >> > to download the files both in YARN and K8S? IMO, this can reduce the
> >> >
> >> > cost
> >> >
> >> > of code maintenance.
> >> >
> >> > Best,
> >> > Weihua
> >> >
> >> >
> >> > On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com>> wrote:
> >> >
> >> > Hi Mason,
> >> >
> >> > Thanks for your input!
> >> >
> >> > +1 for init containers or a more generalized way of obtaining
> >> >
> >> > arbitrary
> >> >
> >> > files. File fetching isn't specific to just SQL--it also matters for
> >> >
> >> > Java
> >> >
> >> > applications if the user doesn't want to rebuild a Flink image and
> >> >
> >> > just
> >> >
> >> > wants to modify the user application fat jar.
> >> >
> >> >
> >> > I agree that utilizing SQL Drivers in Java applications is equally
> >> > important
> >> > as employing them in SQL Gateway. WRT init containers, I think most
> >> > users use them just as a workaround. For example, wget a jar from the
> >> > maven repo.
> >> >
> >> > We could implement the functionality in SQL Driver in a more graceful
> >> > way and the flink-supported filesystem approach seems to be a
> >> > good choice.
> >> >
> >> > Also, what do you think about prefixing the config options with
> >> > `sql-driver` instead of just `sql` to be more specific?
> >> >
> >> >
> >> > LGTM, since SQL Driver is a public interface and the options are
> >> > specific to it.
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> > 2023年6月6日 06:30，Mason Chen <mas.chen6345@gmail.com <mailto:
> mas.chen6345@gmail.com>> 写道：
> >> >
> >> > Hi Paul,
> >> >
> >> > +1 for this feature and supporting SQL file + JSON plans. We get a
> >> >
> >> > lot
> >> >
> >> > of
> >> >
> >> > requests to just be able to submit a SQL file, but the JSON plan
> >> > optimizations make sense.
> >> >
> >> > +1 for init containers or a more generalized way of obtaining
> >> >
> >> > arbitrary
> >> >
> >> > files. File fetching isn't specific to just SQL--it also matters for
> >> >
> >> > Java
> >> >
> >> > applications if the user doesn't want to rebuild a Flink image and
> >> >
> >> > just
> >> >
> >> > wants to modify the user application fat jar.
> >> >
> >> > Please note that we could reuse the checkpoint storage like S3/HDFS,
> >> >
> >> > which
> >> >
> >> > should
> >> >
> >> >
> >> > be required to run Flink in production, so I guess that would be
> >> >
> >> > acceptable
> >> >
> >> > for most
> >> >
> >> >
> >> > users. WDYT?
> >> >
> >> >
> >> > If you do go this route, it would be nice to support writing these
> >> >
> >> > files
> >> >
> >> > to
> >> >
> >> > S3/HDFS via Flink. This makes access control and policy management
> >> >
> >> > simpler.
> >> >
> >> >
> >> > Also, what do you think about prefixing the config options with
> >> > `sql-driver` instead of just `sql` to be more specific?
> >> >
> >> > Best,
> >> > Mason
> >> >
> >> > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com>
> >> >
> >> > <mailto:
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com>>> wrote:
> >> >
> >> >
> >> > Hi Jark,
> >> >
> >> > Thanks for your input! Please see my comments inline.
> >> >
> >> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> >> > DataStream API also doesn't provide a default main class for users,
> >> > why do we need to provide such one for SQL?
> >> >
> >> >
> >> > Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> >> >
> >> > submitted
> >> >
> >> > via Flink CLI which actually could be DataStream/Table jobs.
> >> >
> >> > I think a default main class would be user-friendly which eliminates
> >> >
> >> > the
> >> >
> >> > need
> >> > for users to write a main class as SQLRunner in Flink K8s operator
> >> >
> >> > [1].
> >> >
> >> >
> >> > I thought the proposed SqlDriver was a dedicated main class
> >> >
> >> > accepting
> >> >
> >> > SQL files, is
> >> >
> >> > that correct?
> >> >
> >> >
> >> > Both JSON plans and SQL files are accepted. SQL Gateway should use
> >> >
> >> > JSON
> >> >
> >> > plans,
> >> > while CLI users may use either JSON plans or SQL files.
> >> >
> >> > Please see the updated FLIP[2] for more details.
> >> >
> >> > Personally, I prefer the way of init containers which doesn't
> >> >
> >> > depend
> >> >
> >> > on
> >> >
> >> > additional components.
> >> > This can reduce the moving parts of a production environment.
> >> > Depending on a distributed file system makes the testing, demo, and
> >> >
> >> > local
> >> >
> >> > setup harder than init containers.
> >> >
> >> >
> >> > Please note that we could reuse the checkpoint storage like S3/HDFS,
> >> >
> >> > which
> >> >
> >> > should
> >> > be required to run Flink in production, so I guess that would be
> >> > acceptable for most
> >> > users. WDYT?
> >> >
> >> > WRT testing, demo, and local setups, I think we could support the
> >> >
> >> > local
> >> >
> >> > filesystem
> >> > scheme i.e. file://** <> as the state backends do. It works as long as
> >> >
> >> > SQL
> >> >
> >> > Gateway
> >> > and JobManager(or SQL Driver) can access the resource directory
> >> >
> >> > (specified
> >> >
> >> > via
> >> > `sql-gateway.application.storage-dir`).
> >> >
> >> > Thanks!
> >> >
> >> > [1]
> >> >
> >> >
> >> >
> >> >
> >> >
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> <
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> >
> >> >
> >> > [2]
> >> >
> >> >
> >> >
> >> >
> >> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >
> >> >
> >> > [3]
> >> >
> >> >
> >> >
> >> >
> >> >
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> <
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> >
> >> >
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> > 2023年6月3日 12:21，Jark Wu <imjark@gmail.com <ma...@gmail.com>>
> 写道：
> >> >
> >> > Hi Paul,
> >> >
> >> > Thanks for your reply. I left my comments inline.
> >> >
> >> > As the FLIP said, it’s good to have a default main class for Flink
> >> >
> >> > SQLs,
> >> >
> >> > which allows users to submit Flink SQLs in the same way as
> >> >
> >> > DataStream
> >> >
> >> > jobs, or else users need to write their own main class.
> >> >
> >> >
> >> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> >> > DataStream API also doesn't provide a default main class for users,
> >> > why do we need to provide such one for SQL?
> >> >
> >> > With the help of ExecNodeGraph, do we still need the serialized
> >> > SessionState? If not, we could make SQL Driver accepts two
> >> >
> >> > serialized
> >> >
> >> > formats:
> >> >
> >> >
> >> > No, ExecNodeGraph doesn't need to serialize SessionState. I thought
> >> >
> >> > the
> >> >
> >> > proposed SqlDriver was a dedicated main class accepting SQL files,
> >> >
> >> > is
> >> >
> >> > that correct?
> >> > If true, we have to ship the SessionState for this case which is a
> >> >
> >> > large
> >> >
> >> > work.
> >> > I think we just need a JsonPlanDriver which is a main class that
> >> >
> >> > accepts
> >> >
> >> > JsonPlan as the parameter.
> >> >
> >> >
> >> > The common solutions I know is to use distributed file systems or
> >> >
> >> > use
> >> >
> >> > init containers to localize the resources.
> >> >
> >> >
> >> > Personally, I prefer the way of init containers which doesn't
> >> >
> >> > depend
> >> >
> >> > on
> >> >
> >> > additional components.
> >> > This can reduce the moving parts of a production environment.
> >> > Depending on a distributed file system makes the testing, demo, and
> >> >
> >> > local
> >> >
> >> > setup harder than init containers.
> >> >
> >> > Best,
> >> > Jark
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com>
> >> >
> >> > <mailto:
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com> <mailto:
> paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
> >> >
> >> >
> >> > The FLIP is in the early phase and some details are not included,
> >> >
> >> > but
> >> >
> >> > fortunately, we got lots of valuable ideas from the discussion.
> >> >
> >> > Thanks to everyone who joined the dissuasion!
> >> > @Weihua @Shanmon @Shengkai @Biao @Jark
> >> >
> >> > This weekend I’m gonna revisit and update the FLIP, adding more
> >> > details. Hopefully, we can further align our opinions.
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> > 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com> <mailto:
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com>>> 写道：
> >> >
> >> >
> >> > Hi Jark,
> >> >
> >> > Thanks a lot for your input!
> >> >
> >> > If we decide to submit ExecNodeGraph instead of SQL file, is it
> >> >
> >> > still
> >> >
> >> > necessary to support SQL Driver?
> >> >
> >> >
> >> > I think so. Apart from usage in SQL Gateway, SQL Driver could
> >> >
> >> > simplify
> >> >
> >> > Flink SQL execution with Flink CLI.
> >> >
> >> > As the FLIP said, it’s good to have a default main class for
> >> >
> >> > Flink
> >> >
> >> > SQLs,
> >> >
> >> > which allows users to submit Flink SQLs in the same way as
> >> >
> >> > DataStream
> >> >
> >> > jobs, or else users need to write their own main class.
> >> >
> >> > SQL Driver needs to serialize SessionState which is very
> >> >
> >> > challenging
> >> >
> >> > but not detailed covered in the FLIP.
> >> >
> >> >
> >> > With the help of ExecNodeGraph, do we still need the serialized
> >> > SessionState? If not, we could make SQL Driver accepts two
> >> >
> >> > serialized
> >> >
> >> > formats:
> >> >
> >> > - SQL files for user-facing public usage
> >> > - ExecNodeGraph for internal usage
> >> >
> >> > It’s kind of similar to the relationship between job jars and
> >> >
> >> > jobgraphs.
> >> >
> >> >
> >> > Regarding "K8S doesn't support shipping multiple jars", is that
> >> >
> >> > true?
> >> >
> >> > Is it
> >> >
> >> > possible to support it?
> >> >
> >> >
> >> > Yes, K8s doesn’t distribute any files. It’s the users’
> >> >
> >> > responsibility
> >> >
> >> > to
> >> >
> >> > make
> >> >
> >> > sure the resources are accessible in the containers. The common
> >> >
> >> > solutions
> >> >
> >> > I know is to use distributed file systems or use init containers
> >> >
> >> > to
> >> >
> >> > localize the
> >> >
> >> > resources.
> >> >
> >> > Now I lean toward introducing a fs to do the distribution job.
> >> >
> >> > WDYT?
> >> >
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> > 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com>
> <mailto:
> >> >
> >> > imjark@gmail.com <ma...@gmail.com>>
> >> >
> >> > <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
> imjark@gmail.com <ma...@gmail.com>>>
> >> >
> >> > <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
> imjark@gmail.com <ma...@gmail.com>> <mailto:
> >> >
> >> > imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com
> <ma...@gmail.com>>>>>
> >> >
> >> > 写道：
> >> >
> >> >
> >> > Hi Paul,
> >> >
> >> > Thanks for starting this discussion. I like the proposal! This
> >> >
> >> > is
> >> >
> >> > a
> >> >
> >> > frequently requested feature!
> >> >
> >> > I agree with Shengkai that ExecNodeGraph as the submission
> >> >
> >> > object
> >> >
> >> > is a
> >> >
> >> > better idea than SQL file. To be more specific, it should be
> >> >
> >> > JsonPlanGraph
> >> >
> >> > or CompiledPlan which is the serializable representation.
> >> >
> >> > CompiledPlan
> >> >
> >> > is a
> >> >
> >> > clear separation between compiling/optimization/validation and
> >> >
> >> > execution.
> >> >
> >> > This can keep the validation and metadata accessing still on the
> >> >
> >> > SQLGateway
> >> >
> >> > side. This allows SQLGateway to leverage some metadata caching
> >> >
> >> > and
> >> >
> >> > UDF
> >> >
> >> > JAR
> >> >
> >> > caching for better compiling performance.
> >> >
> >> > If we decide to submit ExecNodeGraph instead of SQL file, is it
> >> >
> >> > still
> >> >
> >> > necessary to support SQL Driver? Regarding non-interactive SQL
> >> >
> >> > jobs,
> >> >
> >> > users
> >> >
> >> > can use the Table API program for application mode. SQL Driver
> >> >
> >> > needs
> >> >
> >> > to
> >> >
> >> > serialize SessionState which is very challenging but not
> >> >
> >> > detailed
> >> >
> >> > covered
> >> >
> >> > in the FLIP.
> >> >
> >> > Regarding "K8S doesn't support shipping multiple jars", is that
> >> >
> >> > true?
> >> >
> >> > Is it
> >> >
> >> > possible to support it?
> >> >
> >> > Best,
> >> > Jark
> >> >
> >> >
> >> >
> >> > On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com>
> >> >
> >> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>
> <mailto:
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com> <mailto:
> paullin3280@gmail.com <ma...@gmail.com>>> <mailto:
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com> <mailto:
> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com> <mailto:
> paullin3280@gmail.com <ma...@gmail.com>>>>> wrote:
> >> >
> >> >
> >> > Hi Weihua,
> >> >
> >> > You’re right. Distributing the SQLs to the TMs is one of the
> >> >
> >> > challenging
> >> >
> >> > parts of this FLIP.
> >> >
> >> > Web submission is not enabled in application mode currently as
> >> >
> >> > you
> >> >
> >> > said,
> >> >
> >> > but it could be changed if we have good reasons.
> >> >
> >> > What do you think about introducing a distributed storage for
> >> >
> >> > SQL
> >> >
> >> > Gateway?
> >> >
> >> >
> >> > We could make use of Flink file systems [1] to distribute the
> >> >
> >> > SQL
> >> >
> >> > Gateway
> >> >
> >> > generated resources, that should solve the problem at its root
> >> >
> >> > cause.
> >> >
> >> >
> >> > Users could specify Flink-supported file systems to ship files.
> >> >
> >> > It’s
> >> >
> >> > only
> >> >
> >> > required when using SQL Gateway with K8s application mode.
> >> >
> >> > [1]
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> >> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> >
> >> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> >> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> > 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> huweihua.ckl@gmail.com> <mailto:
> >> >
> >> > huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
> >> >
> >> > huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
> huweihua.ckl@gmail.com <ma...@gmail.com>>>> 写道：
> >> >
> >> >
> >> > Thanks Paul for your reply.
> >> >
> >> > SQLDriver looks good to me.
> >> >
> >> > 2. Do you mean a pass the SQL string a configuration or a
> >> >
> >> > program
> >> >
> >> > argument?
> >> >
> >> >
> >> >
> >> > I brought this up because we were unable to pass the SQL file
> >> >
> >> > to
> >> >
> >> > Flink
> >> >
> >> > using Kubernetes mode.
> >> > For DataStream/Python users, they need to prepare their images
> >> >
> >> > for
> >> >
> >> > the
> >> >
> >> > jars
> >> >
> >> > and dependencies.
> >> > But for SQL users, they can use a common image to run
> >> >
> >> > different
> >> >
> >> > SQL
> >> >
> >> > queries
> >> >
> >> > if there are no other udf requirements.
> >> > It would be great if the SQL query and image were not bound.
> >> >
> >> > Using strings is a way to decouple these, but just as you
> >> >
> >> > mentioned,
> >> >
> >> > it's
> >> >
> >> > not easy to pass complex SQL.
> >> >
> >> > use web submission
> >> >
> >> > AFAIK, we can not use web submission in the Application mode.
> >> >
> >> > Please
> >> >
> >> > correct me if I'm wrong.
> >> >
> >> >
> >> > Best,
> >> > Weihua
> >> >
> >> >
> >> > On Wed, May 31, 2023 at 9:37 PM Paul Lam <
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com>
> >> >
> >> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>
> >> >
> >> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:
> paullin3280@gmail.com <ma...@gmail.com>>>>
> >> >
> >> > wrote:
> >> >
> >> >
> >> > Hi Biao,
> >> >
> >> > Thanks for your comments!
> >> >
> >> > 1. Scope: is this FLIP only targeted for non-interactive
> >> >
> >> > Flink
> >> >
> >> > SQL
> >> >
> >> > jobs
> >> >
> >> > in
> >> >
> >> > Application mode? More specifically, if we use SQL
> >> >
> >> > client/gateway
> >> >
> >> > to
> >> >
> >> > execute some interactive SQLs like a SELECT query, can we
> >> >
> >> > ask
> >> >
> >> > flink
> >> >
> >> > to
> >> >
> >> > use
> >> >
> >> > Application mode to execute those queries after this FLIP?
> >> >
> >> >
> >> > Thanks for pointing it out. I think only DMLs would be
> >> >
> >> > executed
> >> >
> >> > via
> >> >
> >> > SQL
> >> >
> >> > Driver.
> >> > I'll add the scope to the FLIP.
> >> >
> >> > 2. Deployment: I believe in YARN mode, the implementation is
> >> >
> >> > trivial as
> >> >
> >> > we
> >> >
> >> > can ship files via YARN's tool easily but for K8s, things
> >> >
> >> > can
> >> >
> >> > be
> >> >
> >> > more
> >> >
> >> > complicated as Shengkai said.
> >> >
> >> >
> >> >
> >> > Your input is very informative. I’m thinking about using web
> >> >
> >> > submission,
> >> >
> >> > but it requires exposing the JobManager port which could also
> >> >
> >> > be
> >> >
> >> > a
> >> >
> >> > problem
> >> >
> >> > on K8s.
> >> >
> >> > Another approach is to explicitly require a distributed
> >> >
> >> > storage
> >> >
> >> > to
> >> >
> >> > ship
> >> >
> >> > files,
> >> > but we may need a new deployment executor for that.
> >> >
> >> > What do you think of these two approaches?
> >> >
> >> > 3. Serialization of SessionState: in SessionState, there are
> >> >
> >> > some
> >> >
> >> > unserializable fields
> >> > like
> >> >
> >> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >> >
> >> > It
> >> >
> >> > may be worthwhile to add more details about the
> >> >
> >> > serialization
> >> >
> >> > part.
> >> >
> >> >
> >> > I agree. That’s a missing part. But if we use ExecNodeGraph
> >> >
> >> > as
> >> >
> >> > Shengkai
> >> >
> >> > mentioned, do we eliminate the need for serialization of
> >> >
> >> > SessionState?
> >> >
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> > 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
> biaogeng7@gmail.com> <mailto:
> >> >
> >> > biaogeng7@gmail.com <ma...@gmail.com>> <mailto:
> >> >
> >> > biaogeng7@gmail.com <ma...@gmail.com> <mailto:
> biaogeng7@gmail.com <ma...@gmail.com>>>> 写道：
> >> >
> >> >
> >> > Thanks Paul for the proposal!I believe it would be very
> >> >
> >> > useful
> >> >
> >> > for
> >> >
> >> > flink
> >> >
> >> > users.
> >> > After reading the FLIP, I have some questions:
> >> > 1. Scope: is this FLIP only targeted for non-interactive
> >> >
> >> > Flink
> >> >
> >> > SQL
> >> >
> >> > jobs
> >> >
> >> > in
> >> >
> >> > Application mode? More specifically, if we use SQL
> >> >
> >> > client/gateway
> >> >
> >> > to
> >> >
> >> > execute some interactive SQLs like a SELECT query, can we
> >> >
> >> > ask
> >> >
> >> > flink
> >> >
> >> > to
> >> >
> >> > use
> >> >
> >> > Application mode to execute those queries after this FLIP?
> >> > 2. Deployment: I believe in YARN mode, the implementation is
> >> >
> >> > trivial as
> >> >
> >> > we
> >> >
> >> > can ship files via YARN's tool easily but for K8s, things
> >> >
> >> > can
> >> >
> >> > be
> >> >
> >> > more
> >> >
> >> > complicated as Shengkai said. I have implemented a simple
> >> >
> >> > POC
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> <
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> <
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> >> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> <
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> <
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >> >
> >> >
> >> >
> >> >
> >> > based on SQL client before(i.e. consider the SQL client
> >> >
> >> > which
> >> >
> >> > supports
> >> >
> >> > executing a SQL file as the SQL driver in this FLIP). One
> >> >
> >> > problem
> >> >
> >> > I
> >> >
> >> > have
> >> >
> >> > met is how do we ship SQL files ( or Job Graph) to the k8s
> >> >
> >> > side.
> >> >
> >> > Without
> >> >
> >> > such support, users have to modify the initContainer or
> >> >
> >> > rebuild
> >> >
> >> > a
> >> >
> >> > new
> >> >
> >> > K8s
> >> >
> >> > image every time to fetch the SQL file. Like the flink k8s
> >> >
> >> > operator,
> >> >
> >> > one
> >> >
> >> > workaround is to utilize the flink config(transforming the
> >> >
> >> > SQL
> >> >
> >> > file
> >> >
> >> > to
> >> >
> >> > a
> >> >
> >> > escaped string like Weihua mentioned) which will be
> >> >
> >> > converted
> >> >
> >> > to a
> >> >
> >> > ConfigMap but K8s has size limit of ConfigMaps(no larger
> >> >
> >> > than
> >> >
> >> > 1MB
> >> >
> >> > <
> >> >
> >> > https://kubernetes.io/docs/concepts/configuration/configmap/ <
> https://kubernetes.io/docs/concepts/configuration/configmap/>
> >> >
> >> > <
> >> >
> >> > https://kubernetes.io/docs/concepts/configuration/configmap/ <
> https://kubernetes.io/docs/concepts/configuration/configmap/>> <
> >> >
> >> > https://kubernetes.io/docs/concepts/configuration/configmap/ <
> https://kubernetes.io/docs/concepts/configuration/configmap/> <
> >> >
> >> > https://kubernetes.io/docs/concepts/configuration/configmap/ <
> https://kubernetes.io/docs/concepts/configuration/configmap/>>>>).
> >> >
> >> > Not
> >> >
> >> > sure
> >> >
> >> > if we have better solutions.
> >> > 3. Serialization of SessionState: in SessionState, there are
> >> >
> >> > some
> >> >
> >> > unserializable fields
> >> > like
> >> >
> >> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >> >
> >> > It
> >> >
> >> > may be worthwhile to add more details about the
> >> >
> >> > serialization
> >> >
> >> > part.
> >> >
> >> >
> >> > Best,
> >> > Biao Geng
> >> >
> >> > Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> <mailto:
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com>
> >> >
> >> >
> >> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:
> paullin3280@gmail.com <ma...@gmail.com>>>>
> >> >
> >> > 于2023年5月31日周三 11:49写道：
> >> >
> >> >
> >> > Hi Weihua,
> >> >
> >> > Thanks a lot for your input! Please see my comments inline.
> >> >
> >> > - Is SQLRunner the better name? We use this to run a SQL
> >> >
> >> > Job.
> >> >
> >> > (Not
> >> >
> >> > strong,
> >> >
> >> > the SQLDriver is fine for me)
> >> >
> >> >
> >> > I’ve thought about SQL Runner but picked SQL Driver for the
> >> >
> >> > following
> >> >
> >> > reasons FYI:
> >> >
> >> > 1. I have a PythonDriver doing the same job for PyFlink [1]
> >> > 2. Flink program's main class is sort of like Driver in
> >> >
> >> > JDBC
> >> >
> >> > which
> >> >
> >> > translates SQLs into
> >> > databases specific languages.
> >> >
> >> > In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >> >
> >> > - Could we run SQL jobs using SQL in strings? Otherwise,
> >> >
> >> > we
> >> >
> >> > need
> >> >
> >> > to
> >> >
> >> > prepare
> >> >
> >> > a SQL file in an image for Kubernetes application mode,
> >> >
> >> > which
> >> >
> >> > may
> >> >
> >> > be
> >> >
> >> > a
> >> >
> >> > bit
> >> >
> >> > cumbersome.
> >> >
> >> >
> >> > Do you mean a pass the SQL string a configuration or a
> >> >
> >> > program
> >> >
> >> > argument?
> >> >
> >> >
> >> > I thought it might be convenient for testing propose, but
> >> >
> >> > not
> >> >
> >> > recommended
> >> >
> >> > for production,
> >> > cause Flink SQLs could be complicated and involves lots of
> >> >
> >> > characters
> >> >
> >> > that
> >> >
> >> > need to escape.
> >> >
> >> > WDYT?
> >> >
> >> > - I noticed that we don't specify the SQLDriver jar in the
> >> >
> >> > "run-application"
> >> >
> >> > command. Does that mean we need to perform automatic
> >> >
> >> > detection
> >> >
> >> > in
> >> >
> >> > Flink?
> >> >
> >> >
> >> > Yes! It’s like running a PyFlink job with the following
> >> >
> >> > command:
> >> >
> >> >
> >> > ```
> >> > ./bin/flink run \
> >> > --pyModule table.word_count \
> >> > --pyFiles examples/python/table
> >> > ```
> >> >
> >> > The CLI determines if it’s a SQL job, if yes apply the SQL
> >> >
> >> > Driver
> >> >
> >> > automatically.
> >> >
> >> >
> >> > [1]
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> <
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> <
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> >> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> <
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> <
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >> >
> >> >
> >> >
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> > 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> huweihua.ckl@gmail.com>
> >> >
> >> > <mailto:
> >> >
> >> > huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
> >> >
> >> > huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
> >> >
> >> >
> >> > Thanks Paul for the proposal.
> >> >
> >> > +1 for this. It is valuable in improving ease of use.
> >> >
> >> > I have a few questions.
> >> > - Is SQLRunner the better name? We use this to run a SQL
> >> >
> >> > Job.
> >> >
> >> > (Not
> >> >
> >> > strong,
> >> >
> >> > the SQLDriver is fine for me)
> >> > - Could we run SQL jobs using SQL in strings? Otherwise,
> >> >
> >> > we
> >> >
> >> > need
> >> >
> >> > to
> >> >
> >> > prepare
> >> >
> >> > a SQL file in an image for Kubernetes application mode,
> >> >
> >> > which
> >> >
> >> > may
> >> >
> >> > be
> >> >
> >> > a
> >> >
> >> > bit
> >> >
> >> > cumbersome.
> >> > - I noticed that we don't specify the SQLDriver jar in the
> >> >
> >> > "run-application"
> >> >
> >> > command. Does that mean we need to perform automatic
> >> >
> >> > detection
> >> >
> >> > in
> >> >
> >> > Flink?
> >> >
> >> >
> >> >
> >> > Best,
> >> > Weihua
> >> >
> >> >
> >> > On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> >> >
> >> > paullin3280@gmail.com <ma...@gmail.com>
> >> >
> >> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:
> paullin3280@gmail.com <ma...@gmail.com>>>>
> >> >
> >> > wrote:
> >> >
> >> >
> >> > Hi team,
> >> >
> >> > I’d like to start a discussion about FLIP-316 [1], which
> >> >
> >> > introduces
> >> >
> >> > a
> >> >
> >> > SQL
> >> >
> >> > driver as the
> >> > default main class for Flink SQL jobs.
> >> >
> >> > Currently, Flink SQL could be executed out of the box
> >> >
> >> > either
> >> >
> >> > via
> >> >
> >> > SQL
> >> >
> >> > Client/Gateway
> >> > or embedded in a Flink Java/Python program.
> >> >
> >> > However, each one has its drawback:
> >> >
> >> > - SQL Client/Gateway doesn’t support the application
> >> >
> >> > deployment
> >> >
> >> > mode
> >> >
> >> > [2]
> >> >
> >> > - Flink Java/Python program requires extra work to write
> >> >
> >> > a
> >> >
> >> > non-SQL
> >> >
> >> > program
> >> >
> >> >
> >> > Therefore, I propose adding a SQL driver to act as the
> >> >
> >> > default
> >> >
> >> > main
> >> >
> >> > class
> >> >
> >> > for SQL jobs.
> >> > Please see the FLIP docs for details and feel free to
> >> >
> >> > comment.
> >> >
> >> > Thanks!
> >> >
> >> >
> >> > [1]
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >
> >> >
> >> > <
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >
> >> >
> >> >
> >> > [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> https://issues.apache.org/jira/browse/FLINK-26541> <
> >> > https://issues.apache.org/jira/browse/FLINK-26541 <
> https://issues.apache.org/jira/browse/FLINK-26541>>
> >> >
> >> > Best,
> >> > Paul Lam
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Shengkai,
> * How can we ship the json plan to the JobManager?
The Flink K8s module should be responsible for file distribution. We could introduce
an option like `kubernetes.storage.dir`. For each flink cluster, there would be a
dedicated subdirectory, with the pattern like `${kubernetes.storage.dir}/${cluster-id}`.
All resources-related options (e.g. pipeline jars, json plans) that are configured with
scheme `file://` <file:///%60> would be uploaded to the resource directory and downloaded to the
jobmanager, before SQL Driver accesses the files with the original filenames.
> * Classloading strategy
We could directly specify the SQL Gateway jar as the jar file in PackagedProgram.
It would be treated like a normal user jar and the SQL Driver is loaded into the user
classloader. WDYT?
> * Option `$internal.sql-gateway.driver.sql-config` is string type
> I think it's better to use Map type here
By Map type configuration, do you mean a nested map that contains all
configurations?
I hope I've explained myself well, it’s a file that contains the extra SQL
configurations, which would be shipped to the jobmanager.
> * PoC branch
Sure. I’ll let you know once I get the job done.
Best,
Paul Lam
> 2023年6月26日 14:27，Shengkai Fang <fs...@gmail.com> 写道：
>
> Hi, Paul.
>
> Thanks for your update. I have a few questions about the new design:
>
> * How can we ship the json plan to the JobManager?
>
> The current design only exposes an option about the URL of the json plan. It seems the gateway is responsible to upload to an external stroage. Can we reuse the PipelineOptions.JARS to ship to the remote filesystem?
>
> * Classloading strategy
>
> Currently, the Driver is in the sql-gateway package. It means the Driver is not in the JM's classpath directly. Because the sql-gateway jar is now in the opt directory rather than lib directory. It may need to add the external dependencies as Python does[1]. BTW, I think it's better to move the Driver into the flink-table-runtime package, which is much easier to find(Sorry for the wrong opinion before).
>
> * Option `$internal.sql-gateway.driver.sql-config` is string type
>
> I think it's better to use Map type here
>
> * PoC branch
>
> Because this FLIP involves many modules, do you have a PoC branch to verify it does work?
>
> Best,
> Shengkai
>
> [1] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 <https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940>
> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> 于2023年6月19日周一 14:09写道：
> Hi Shengkai,
>
> Sorry for my late reply. It took me some time to update the FLIP.
>
> In the latest FLIP design, SQL Driver is placed in flink-sql-gateway module. PTAL.
>
> The FLIP does not cover details about the K8s file distribution, but its general usage would
> be very much the same as YARN setups. We could make follow-up discussions in the jira
> tickets.
>
> Best,
> Paul Lam
>
>> 2023年6月12日 15:29，Shengkai Fang <fskmine@gmail.com <ma...@gmail.com>> 写道：
>>
>>
>> > If it’s the case, I’m good with introducing a new module and making SQL Driver
>> > an internal class and accepts JSON plans only.
>>
>> I rethink this again and again. I think it's better to move the SqlDriver into the sql-gateway module because the sql client relies on the sql-gateway to submit the sql and the sql-gateway has the ability to generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
>>
>> * Upload configuration through command line parameter
>>
>> ExecNodeGraph only contains the job's information but it doesn't contain the checkpoint dir, checkpoint interval, execution mode and so on. So I think we should also upload the configuration.
>>
>> * KubernetesClusterDescripter and KubernetesApplicationClusterEntrypoint are responsible for the jar upload/download
>>
>> +1 for the change.
>>
>> Could you update the FLIP about the current discussion?
>>
>> Best,
>> Shengkai
>>
>>
>>
>>
>>
>>
>> Yang Wang <wangyang0918@apache.org <ma...@apache.org>> 于2023年6月12日周一 11:41写道：
>> Sorry for the late reply. I am in favor of introducing such a built-in
>> resource localization mechanism
>> based on Flink FileSystem. Then FLINK-28915[1] could be the second step
>> which will download
>> the jars and dependencies to the JobManager/TaskManager local directory
>> before working.
>>
>> The first step could be done in another ticket in Flink. Or some external
>> Flink jobs management system
>> could also take care of this.
>>
>> [1]. https://issues.apache.org/jira/browse/FLINK-28915 <https://issues.apache.org/jira/browse/FLINK-28915>
>>
>> Best,
>> Yang
>>
>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> 于2023年6月9日周五 17:39写道：
>>
>> > Hi Mason,
>> >
>> > I get your point. I'm increasingly feeling the need to introduce a
>> > built-in
>> > file distribution mechanism for flink-kubernetes module, just like Spark
>> > does with `spark.kubernetes.file.upload.path` [1].
>> >
>> > I’m assuming the workflow is as follows:
>> >
>> > - KubernetesClusterDescripter uploads all local resources to a remote
>> > storage via Flink filesystem (skips if the resources are already remote).
>> > - KubernetesApplicationClusterEntrypoint downloads the resources
>> > and put them in the classpath during startup.
>> >
>> > I wouldn't mind splitting it into another FLIP to ensure that everything is
>> > done correctly.
>> >
>> > cc'ed @Yang to gather more opinions.
>> >
>> > [1]
>> > https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management <https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management>
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月8日 12:15，Mason Chen <mas.chen6345@gmail.com <ma...@gmail.com>> 写道：
>> >
>> > Hi Paul,
>> >
>> > Thanks for your response!
>> >
>> > I agree that utilizing SQL Drivers in Java applications is equally
>> > important
>> >
>> > as employing them in SQL Gateway. WRT init containers, I think most
>> > users use them just as a workaround. For example, wget a jar from the
>> > maven repo.
>> >
>> > We could implement the functionality in SQL Driver in a more graceful
>> > way and the flink-supported filesystem approach seems to be a
>> > good choice.
>> >
>> >
>> > My main point is: can we solve the problem with a design agnostic of SQL
>> > and Stream API? I mentioned a use case where this ability is useful for
>> > Java or Stream API applications. Maybe this is even a non-goal to your FLIP
>> > since you are focusing on the driver entrypoint.
>> >
>> > Jark mentioned some optimizations:
>> >
>> > This allows SQLGateway to leverage some metadata caching and UDF JAR
>> > caching for better compiling performance.
>> >
>> > It would be great to see this even outside the SQLGateway (i.e. UDF JAR
>> > caching).
>> >
>> > Best,
>> > Mason
>> >
>> > On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskmine@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Hi. Paul. Thanks for your update and the update makes me understand the
>> > design much better.
>> >
>> > But I still have some questions about the FLIP.
>> >
>> > For SQL Gateway, only DMLs need to be delegated to the SQL server
>> > Driver. I would think about the details and update the FLIP. Do you have
>> >
>> > some
>> >
>> > ideas already?
>> >
>> >
>> > If the applicaiton mode can not support library mode, I think we should
>> > only execute INSERT INTO and UPDATE/ DELETE statement in the application
>> > mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
>> > statements. The ANALYZE TABLE syntax need to register the statistic to the
>> > catalog after job finishes and the CALL PROCEDURE statement doesn't
>> > generate the ExecNodeGraph.
>> >
>> > * Introduce storage via option `sql-gateway.application.storage-dir`
>> >
>> > If we can not support to submit the jars through web submission, +1 to
>> > introduce the options to upload the files. While I think the uploader
>> > should be responsible to remove the uploaded jars. Can we remove the jars
>> > if the job is running or gateway exits?
>> >
>> > * JobID is not avaliable
>> >
>> > Can we use the returned rest client by ApplicationDeployer to query the job
>> > id? I am concerned that users don't know which job is related to the
>> > submitted SQL.
>> >
>> > * Do we need to introduce a new module named flink-table-sql-runner?
>> >
>> > It seems we need to introduce a new module. Will the new module is
>> > available in the distribution package? I agree with Jark that we don't need
>> > to introduce this for table-API users and these users have their main
>> > class. If we want to make users write the k8s operator more easily, I think
>> > we should modify the k8s operator repo. If we don't need to support SQL
>> > files, can we make this jar only visible in the sql-gateway like we do in
>> > the planner loader?[1]
>> >
>> > [1]
>> >
>> >
>> > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 <https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95>
>> >
>> > Best,
>> > Shengkai
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com>> 于2023年6月7日周三 10:52写道：
>> >
>> > Hi,
>> >
>> > Thanks for updating the FLIP.
>> >
>> > I have two cents on the distribution of SQLs and resources.
>> > 1. Should we support a common file distribution mechanism for k8s
>> > application mode?
>> > I have seen some issues and requirements on the mailing list.
>> > In our production environment, we implement the download command in the
>> > CliFrontend.
>> > And automatically add an init container to the POD for file
>> >
>> > downloading.
>> >
>> > The advantage of this
>> > is that we can use all Flink-supported file systems to store files.
>> >
>> > This need more discussion. I would appreciate hearing more opinions.
>> >
>> > 2. In this FLIP, we distribute files in two different ways in YARN and
>> > Kubernetes. Can we combine it in one way?
>> > If we don't want to implement a common file distribution for k8s
>> > application mode. Could we use the SQLDriver
>> > to download the files both in YARN and K8S? IMO, this can reduce the
>> >
>> > cost
>> >
>> > of code maintenance.
>> >
>> > Best,
>> > Weihua
>> >
>> >
>> > On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Hi Mason,
>> >
>> > Thanks for your input!
>> >
>> > +1 for init containers or a more generalized way of obtaining
>> >
>> > arbitrary
>> >
>> > files. File fetching isn't specific to just SQL--it also matters for
>> >
>> > Java
>> >
>> > applications if the user doesn't want to rebuild a Flink image and
>> >
>> > just
>> >
>> > wants to modify the user application fat jar.
>> >
>> >
>> > I agree that utilizing SQL Drivers in Java applications is equally
>> > important
>> > as employing them in SQL Gateway. WRT init containers, I think most
>> > users use them just as a workaround. For example, wget a jar from the
>> > maven repo.
>> >
>> > We could implement the functionality in SQL Driver in a more graceful
>> > way and the flink-supported filesystem approach seems to be a
>> > good choice.
>> >
>> > Also, what do you think about prefixing the config options with
>> > `sql-driver` instead of just `sql` to be more specific?
>> >
>> >
>> > LGTM, since SQL Driver is a public interface and the options are
>> > specific to it.
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月6日 06:30，Mason Chen <mas.chen6345@gmail.com <ma...@gmail.com>> 写道：
>> >
>> > Hi Paul,
>> >
>> > +1 for this feature and supporting SQL file + JSON plans. We get a
>> >
>> > lot
>> >
>> > of
>> >
>> > requests to just be able to submit a SQL file, but the JSON plan
>> > optimizations make sense.
>> >
>> > +1 for init containers or a more generalized way of obtaining
>> >
>> > arbitrary
>> >
>> > files. File fetching isn't specific to just SQL--it also matters for
>> >
>> > Java
>> >
>> > applications if the user doesn't want to rebuild a Flink image and
>> >
>> > just
>> >
>> > wants to modify the user application fat jar.
>> >
>> > Please note that we could reuse the checkpoint storage like S3/HDFS,
>> >
>> > which
>> >
>> > should
>> >
>> >
>> > be required to run Flink in production, so I guess that would be
>> >
>> > acceptable
>> >
>> > for most
>> >
>> >
>> > users. WDYT?
>> >
>> >
>> > If you do go this route, it would be nice to support writing these
>> >
>> > files
>> >
>> > to
>> >
>> > S3/HDFS via Flink. This makes access control and policy management
>> >
>> > simpler.
>> >
>> >
>> > Also, what do you think about prefixing the config options with
>> > `sql-driver` instead of just `sql` to be more specific?
>> >
>> > Best,
>> > Mason
>> >
>> > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
>> >
>> > <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>> >
>> >
>> > Hi Jark,
>> >
>> > Thanks for your input! Please see my comments inline.
>> >
>> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>> > DataStream API also doesn't provide a default main class for users,
>> > why do we need to provide such one for SQL?
>> >
>> >
>> > Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>> >
>> > submitted
>> >
>> > via Flink CLI which actually could be DataStream/Table jobs.
>> >
>> > I think a default main class would be user-friendly which eliminates
>> >
>> > the
>> >
>> > need
>> > for users to write a main class as SQLRunner in Flink K8s operator
>> >
>> > [1].
>> >
>> >
>> > I thought the proposed SqlDriver was a dedicated main class
>> >
>> > accepting
>> >
>> > SQL files, is
>> >
>> > that correct?
>> >
>> >
>> > Both JSON plans and SQL files are accepted. SQL Gateway should use
>> >
>> > JSON
>> >
>> > plans,
>> > while CLI users may use either JSON plans or SQL files.
>> >
>> > Please see the updated FLIP[2] for more details.
>> >
>> > Personally, I prefer the way of init containers which doesn't
>> >
>> > depend
>> >
>> > on
>> >
>> > additional components.
>> > This can reduce the moving parts of a production environment.
>> > Depending on a distributed file system makes the testing, demo, and
>> >
>> > local
>> >
>> > setup harder than init containers.
>> >
>> >
>> > Please note that we could reuse the checkpoint storage like S3/HDFS,
>> >
>> > which
>> >
>> > should
>> > be required to run Flink in production, so I guess that would be
>> > acceptable for most
>> > users. WDYT?
>> >
>> > WRT testing, demo, and local setups, I think we could support the
>> >
>> > local
>> >
>> > filesystem
>> > scheme i.e. file://** <> as the state backends do. It works as long as
>> >
>> > SQL
>> >
>> > Gateway
>> > and JobManager(or SQL Driver) can access the resource directory
>> >
>> > (specified
>> >
>> > via
>> > `sql-gateway.application.storage-dir`).
>> >
>> > Thanks!
>> >
>> > [1]
>> >
>> >
>> >
>> >
>> > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java <https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java>
>> >
>> > [2]
>> >
>> >
>> >
>> >
>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
>> >
>> > [3]
>> >
>> >
>> >
>> >
>> > https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 <https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161>
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月3日 12:21，Jark Wu <imjark@gmail.com <ma...@gmail.com>> 写道：
>> >
>> > Hi Paul,
>> >
>> > Thanks for your reply. I left my comments inline.
>> >
>> > As the FLIP said, it’s good to have a default main class for Flink
>> >
>> > SQLs,
>> >
>> > which allows users to submit Flink SQLs in the same way as
>> >
>> > DataStream
>> >
>> > jobs, or else users need to write their own main class.
>> >
>> >
>> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>> > DataStream API also doesn't provide a default main class for users,
>> > why do we need to provide such one for SQL?
>> >
>> > With the help of ExecNodeGraph, do we still need the serialized
>> > SessionState? If not, we could make SQL Driver accepts two
>> >
>> > serialized
>> >
>> > formats:
>> >
>> >
>> > No, ExecNodeGraph doesn't need to serialize SessionState. I thought
>> >
>> > the
>> >
>> > proposed SqlDriver was a dedicated main class accepting SQL files,
>> >
>> > is
>> >
>> > that correct?
>> > If true, we have to ship the SessionState for this case which is a
>> >
>> > large
>> >
>> > work.
>> > I think we just need a JsonPlanDriver which is a main class that
>> >
>> > accepts
>> >
>> > JsonPlan as the parameter.
>> >
>> >
>> > The common solutions I know is to use distributed file systems or
>> >
>> > use
>> >
>> > init containers to localize the resources.
>> >
>> >
>> > Personally, I prefer the way of init containers which doesn't
>> >
>> > depend
>> >
>> > on
>> >
>> > additional components.
>> > This can reduce the moving parts of a production environment.
>> > Depending on a distributed file system makes the testing, demo, and
>> >
>> > local
>> >
>> > setup harder than init containers.
>> >
>> > Best,
>> > Jark
>> >
>> >
>> >
>> >
>> > On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
>> >
>> > <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>> >
>> >
>> > The FLIP is in the early phase and some details are not included,
>> >
>> > but
>> >
>> > fortunately, we got lots of valuable ideas from the discussion.
>> >
>> > Thanks to everyone who joined the dissuasion!
>> > @Weihua @Shanmon @Shengkai @Biao @Jark
>> >
>> > This weekend I’m gonna revisit and update the FLIP, adding more
>> > details. Hopefully, we can further align our opinions.
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>>> 写道：
>> >
>> >
>> > Hi Jark,
>> >
>> > Thanks a lot for your input!
>> >
>> > If we decide to submit ExecNodeGraph instead of SQL file, is it
>> >
>> > still
>> >
>> > necessary to support SQL Driver?
>> >
>> >
>> > I think so. Apart from usage in SQL Gateway, SQL Driver could
>> >
>> > simplify
>> >
>> > Flink SQL execution with Flink CLI.
>> >
>> > As the FLIP said, it’s good to have a default main class for
>> >
>> > Flink
>> >
>> > SQLs,
>> >
>> > which allows users to submit Flink SQLs in the same way as
>> >
>> > DataStream
>> >
>> > jobs, or else users need to write their own main class.
>> >
>> > SQL Driver needs to serialize SessionState which is very
>> >
>> > challenging
>> >
>> > but not detailed covered in the FLIP.
>> >
>> >
>> > With the help of ExecNodeGraph, do we still need the serialized
>> > SessionState? If not, we could make SQL Driver accepts two
>> >
>> > serialized
>> >
>> > formats:
>> >
>> > - SQL files for user-facing public usage
>> > - ExecNodeGraph for internal usage
>> >
>> > It’s kind of similar to the relationship between job jars and
>> >
>> > jobgraphs.
>> >
>> >
>> > Regarding "K8S doesn't support shipping multiple jars", is that
>> >
>> > true?
>> >
>> > Is it
>> >
>> > possible to support it?
>> >
>> >
>> > Yes, K8s doesn’t distribute any files. It’s the users’
>> >
>> > responsibility
>> >
>> > to
>> >
>> > make
>> >
>> > sure the resources are accessible in the containers. The common
>> >
>> > solutions
>> >
>> > I know is to use distributed file systems or use init containers
>> >
>> > to
>> >
>> > localize the
>> >
>> > resources.
>> >
>> > Now I lean toward introducing a fs to do the distribution job.
>> >
>> > WDYT?
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com> <mailto:
>> >
>> > imjark@gmail.com <ma...@gmail.com>>
>> >
>> > <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>
>> >
>> > <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>> <mailto:
>> >
>> > imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>>>
>> >
>> > 写道：
>> >
>> >
>> > Hi Paul,
>> >
>> > Thanks for starting this discussion. I like the proposal! This
>> >
>> > is
>> >
>> > a
>> >
>> > frequently requested feature!
>> >
>> > I agree with Shengkai that ExecNodeGraph as the submission
>> >
>> > object
>> >
>> > is a
>> >
>> > better idea than SQL file. To be more specific, it should be
>> >
>> > JsonPlanGraph
>> >
>> > or CompiledPlan which is the serializable representation.
>> >
>> > CompiledPlan
>> >
>> > is a
>> >
>> > clear separation between compiling/optimization/validation and
>> >
>> > execution.
>> >
>> > This can keep the validation and metadata accessing still on the
>> >
>> > SQLGateway
>> >
>> > side. This allows SQLGateway to leverage some metadata caching
>> >
>> > and
>> >
>> > UDF
>> >
>> > JAR
>> >
>> > caching for better compiling performance.
>> >
>> > If we decide to submit ExecNodeGraph instead of SQL file, is it
>> >
>> > still
>> >
>> > necessary to support SQL Driver? Regarding non-interactive SQL
>> >
>> > jobs,
>> >
>> > users
>> >
>> > can use the Table API program for application mode. SQL Driver
>> >
>> > needs
>> >
>> > to
>> >
>> > serialize SessionState which is very challenging but not
>> >
>> > detailed
>> >
>> > covered
>> >
>> > in the FLIP.
>> >
>> > Regarding "K8S doesn't support shipping multiple jars", is that
>> >
>> > true?
>> >
>> > Is it
>> >
>> > possible to support it?
>> >
>> > Best,
>> > Jark
>> >
>> >
>> >
>> > On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
>> >
>> > <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>> wrote:
>> >
>> >
>> > Hi Weihua,
>> >
>> > You’re right. Distributing the SQLs to the TMs is one of the
>> >
>> > challenging
>> >
>> > parts of this FLIP.
>> >
>> > Web submission is not enabled in application mode currently as
>> >
>> > you
>> >
>> > said,
>> >
>> > but it could be changed if we have good reasons.
>> >
>> > What do you think about introducing a distributed storage for
>> >
>> > SQL
>> >
>> > Gateway?
>> >
>> >
>> > We could make use of Flink file systems [1] to distribute the
>> >
>> > SQL
>> >
>> > Gateway
>> >
>> > generated resources, that should solve the problem at its root
>> >
>> > cause.
>> >
>> >
>> > Users could specify Flink-supported file systems to ship files.
>> >
>> > It’s
>> >
>> > only
>> >
>> > required when using SQL Gateway with K8s application mode.
>> >
>> > [1]
>> >
>> >
>> >
>> >
>> >
>> >
>> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> >
>> > <
>> >
>> >
>> >
>> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> >
>> > <
>> >
>> >
>> >
>> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> >
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> >
>> > <
>> >
>> >
>> >
>> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> >
>> > <
>> >
>> >
>> >
>> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> >
>> >
>> >
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
>> >
>> > huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
>> >
>> > huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>>>> 写道：
>> >
>> >
>> > Thanks Paul for your reply.
>> >
>> > SQLDriver looks good to me.
>> >
>> > 2. Do you mean a pass the SQL string a configuration or a
>> >
>> > program
>> >
>> > argument?
>> >
>> >
>> >
>> > I brought this up because we were unable to pass the SQL file
>> >
>> > to
>> >
>> > Flink
>> >
>> > using Kubernetes mode.
>> > For DataStream/Python users, they need to prepare their images
>> >
>> > for
>> >
>> > the
>> >
>> > jars
>> >
>> > and dependencies.
>> > But for SQL users, they can use a common image to run
>> >
>> > different
>> >
>> > SQL
>> >
>> > queries
>> >
>> > if there are no other udf requirements.
>> > It would be great if the SQL query and image were not bound.
>> >
>> > Using strings is a way to decouple these, but just as you
>> >
>> > mentioned,
>> >
>> > it's
>> >
>> > not easy to pass complex SQL.
>> >
>> > use web submission
>> >
>> > AFAIK, we can not use web submission in the Application mode.
>> >
>> > Please
>> >
>> > correct me if I'm wrong.
>> >
>> >
>> > Best,
>> > Weihua
>> >
>> >
>> > On Wed, May 31, 2023 at 9:37 PM Paul Lam <
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>
>> >
>> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>
>> >
>> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
>> >
>> > wrote:
>> >
>> >
>> > Hi Biao,
>> >
>> > Thanks for your comments!
>> >
>> > 1. Scope: is this FLIP only targeted for non-interactive
>> >
>> > Flink
>> >
>> > SQL
>> >
>> > jobs
>> >
>> > in
>> >
>> > Application mode? More specifically, if we use SQL
>> >
>> > client/gateway
>> >
>> > to
>> >
>> > execute some interactive SQLs like a SELECT query, can we
>> >
>> > ask
>> >
>> > flink
>> >
>> > to
>> >
>> > use
>> >
>> > Application mode to execute those queries after this FLIP?
>> >
>> >
>> > Thanks for pointing it out. I think only DMLs would be
>> >
>> > executed
>> >
>> > via
>> >
>> > SQL
>> >
>> > Driver.
>> > I'll add the scope to the FLIP.
>> >
>> > 2. Deployment: I believe in YARN mode, the implementation is
>> >
>> > trivial as
>> >
>> > we
>> >
>> > can ship files via YARN's tool easily but for K8s, things
>> >
>> > can
>> >
>> > be
>> >
>> > more
>> >
>> > complicated as Shengkai said.
>> >
>> >
>> >
>> > Your input is very informative. I’m thinking about using web
>> >
>> > submission,
>> >
>> > but it requires exposing the JobManager port which could also
>> >
>> > be
>> >
>> > a
>> >
>> > problem
>> >
>> > on K8s.
>> >
>> > Another approach is to explicitly require a distributed
>> >
>> > storage
>> >
>> > to
>> >
>> > ship
>> >
>> > files,
>> > but we may need a new deployment executor for that.
>> >
>> > What do you think of these two approaches?
>> >
>> > 3. Serialization of SessionState: in SessionState, there are
>> >
>> > some
>> >
>> > unserializable fields
>> > like
>> >
>> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
>> >
>> > It
>> >
>> > may be worthwhile to add more details about the
>> >
>> > serialization
>> >
>> > part.
>> >
>> >
>> > I agree. That’s a missing part. But if we use ExecNodeGraph
>> >
>> > as
>> >
>> > Shengkai
>> >
>> > mentioned, do we eliminate the need for serialization of
>> >
>> > SessionState?
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <ma...@gmail.com> <mailto:
>> >
>> > biaogeng7@gmail.com <ma...@gmail.com>> <mailto:
>> >
>> > biaogeng7@gmail.com <ma...@gmail.com> <mailto:biaogeng7@gmail.com <ma...@gmail.com>>>> 写道：
>> >
>> >
>> > Thanks Paul for the proposal!I believe it would be very
>> >
>> > useful
>> >
>> > for
>> >
>> > flink
>> >
>> > users.
>> > After reading the FLIP, I have some questions:
>> > 1. Scope: is this FLIP only targeted for non-interactive
>> >
>> > Flink
>> >
>> > SQL
>> >
>> > jobs
>> >
>> > in
>> >
>> > Application mode? More specifically, if we use SQL
>> >
>> > client/gateway
>> >
>> > to
>> >
>> > execute some interactive SQLs like a SELECT query, can we
>> >
>> > ask
>> >
>> > flink
>> >
>> > to
>> >
>> > use
>> >
>> > Application mode to execute those queries after this FLIP?
>> > 2. Deployment: I believe in YARN mode, the implementation is
>> >
>> > trivial as
>> >
>> > we
>> >
>> > can ship files via YARN's tool easily but for K8s, things
>> >
>> > can
>> >
>> > be
>> >
>> > more
>> >
>> > complicated as Shengkai said. I have implemented a simple
>> >
>> > POC
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> >
>> > <
>> >
>> >
>> >
>> > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> >
>> > <
>> >
>> >
>> >
>> > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> >
>> >
>> >
>> >
>> > based on SQL client before(i.e. consider the SQL client
>> >
>> > which
>> >
>> > supports
>> >
>> > executing a SQL file as the SQL driver in this FLIP). One
>> >
>> > problem
>> >
>> > I
>> >
>> > have
>> >
>> > met is how do we ship SQL files ( or Job Graph) to the k8s
>> >
>> > side.
>> >
>> > Without
>> >
>> > such support, users have to modify the initContainer or
>> >
>> > rebuild
>> >
>> > a
>> >
>> > new
>> >
>> > K8s
>> >
>> > image every time to fetch the SQL file. Like the flink k8s
>> >
>> > operator,
>> >
>> > one
>> >
>> > workaround is to utilize the flink config(transforming the
>> >
>> > SQL
>> >
>> > file
>> >
>> > to
>> >
>> > a
>> >
>> > escaped string like Weihua mentioned) which will be
>> >
>> > converted
>> >
>> > to a
>> >
>> > ConfigMap but K8s has size limit of ConfigMaps(no larger
>> >
>> > than
>> >
>> > 1MB
>> >
>> > <
>> >
>> > https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>
>> >
>> > <
>> >
>> > https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>> <
>> >
>> > https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> >
>> > https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>>>>).
>> >
>> > Not
>> >
>> > sure
>> >
>> > if we have better solutions.
>> > 3. Serialization of SessionState: in SessionState, there are
>> >
>> > some
>> >
>> > unserializable fields
>> > like
>> >
>> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
>> >
>> > It
>> >
>> > may be worthwhile to add more details about the
>> >
>> > serialization
>> >
>> > part.
>> >
>> >
>> > Best,
>> > Biao Geng
>> >
>> > Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>
>> >
>> >
>> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
>> >
>> > 于2023年5月31日周三 11:49写道：
>> >
>> >
>> > Hi Weihua,
>> >
>> > Thanks a lot for your input! Please see my comments inline.
>> >
>> > - Is SQLRunner the better name? We use this to run a SQL
>> >
>> > Job.
>> >
>> > (Not
>> >
>> > strong,
>> >
>> > the SQLDriver is fine for me)
>> >
>> >
>> > I’ve thought about SQL Runner but picked SQL Driver for the
>> >
>> > following
>> >
>> > reasons FYI:
>> >
>> > 1. I have a PythonDriver doing the same job for PyFlink [1]
>> > 2. Flink program's main class is sort of like Driver in
>> >
>> > JDBC
>> >
>> > which
>> >
>> > translates SQLs into
>> > databases specific languages.
>> >
>> > In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>> >
>> > - Could we run SQL jobs using SQL in strings? Otherwise,
>> >
>> > we
>> >
>> > need
>> >
>> > to
>> >
>> > prepare
>> >
>> > a SQL file in an image for Kubernetes application mode,
>> >
>> > which
>> >
>> > may
>> >
>> > be
>> >
>> > a
>> >
>> > bit
>> >
>> > cumbersome.
>> >
>> >
>> > Do you mean a pass the SQL string a configuration or a
>> >
>> > program
>> >
>> > argument?
>> >
>> >
>> > I thought it might be convenient for testing propose, but
>> >
>> > not
>> >
>> > recommended
>> >
>> > for production,
>> > cause Flink SQLs could be complicated and involves lots of
>> >
>> > characters
>> >
>> > that
>> >
>> > need to escape.
>> >
>> > WDYT?
>> >
>> > - I noticed that we don't specify the SQLDriver jar in the
>> >
>> > "run-application"
>> >
>> > command. Does that mean we need to perform automatic
>> >
>> > detection
>> >
>> > in
>> >
>> > Flink?
>> >
>> >
>> > Yes! It’s like running a PyFlink job with the following
>> >
>> > command:
>> >
>> >
>> > ```
>> > ./bin/flink run \
>> > --pyModule table.word_count \
>> > --pyFiles examples/python/table
>> > ```
>> >
>> > The CLI determines if it’s a SQL job, if yes apply the SQL
>> >
>> > Driver
>> >
>> > automatically.
>> >
>> >
>> > [1]
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> >
>> > <
>> >
>> >
>> >
>> > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> >
>> > <
>> >
>> >
>> >
>> > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> >
>> >
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com>
>> >
>> > <mailto:
>> >
>> > huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
>> >
>> > huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
>> >
>> >
>> > Thanks Paul for the proposal.
>> >
>> > +1 for this. It is valuable in improving ease of use.
>> >
>> > I have a few questions.
>> > - Is SQLRunner the better name? We use this to run a SQL
>> >
>> > Job.
>> >
>> > (Not
>> >
>> > strong,
>> >
>> > the SQLDriver is fine for me)
>> > - Could we run SQL jobs using SQL in strings? Otherwise,
>> >
>> > we
>> >
>> > need
>> >
>> > to
>> >
>> > prepare
>> >
>> > a SQL file in an image for Kubernetes application mode,
>> >
>> > which
>> >
>> > may
>> >
>> > be
>> >
>> > a
>> >
>> > bit
>> >
>> > cumbersome.
>> > - I noticed that we don't specify the SQLDriver jar in the
>> >
>> > "run-application"
>> >
>> > command. Does that mean we need to perform automatic
>> >
>> > detection
>> >
>> > in
>> >
>> > Flink?
>> >
>> >
>> >
>> > Best,
>> > Weihua
>> >
>> >
>> > On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>
>> >
>> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
>> >
>> > wrote:
>> >
>> >
>> > Hi team,
>> >
>> > I’d like to start a discussion about FLIP-316 [1], which
>> >
>> > introduces
>> >
>> > a
>> >
>> > SQL
>> >
>> > driver as the
>> > default main class for Flink SQL jobs.
>> >
>> > Currently, Flink SQL could be executed out of the box
>> >
>> > either
>> >
>> > via
>> >
>> > SQL
>> >
>> > Client/Gateway
>> > or embedded in a Flink Java/Python program.
>> >
>> > However, each one has its drawback:
>> >
>> > - SQL Client/Gateway doesn’t support the application
>> >
>> > deployment
>> >
>> > mode
>> >
>> > [2]
>> >
>> > - Flink Java/Python program requires extra work to write
>> >
>> > a
>> >
>> > non-SQL
>> >
>> > program
>> >
>> >
>> > Therefore, I propose adding a SQL driver to act as the
>> >
>> > default
>> >
>> > main
>> >
>> > class
>> >
>> > for SQL jobs.
>> > Please see the FLIP docs for details and feel free to
>> >
>> > comment.
>> >
>> > Thanks!
>> >
>> >
>> > [1]
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver>
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
>> >
>> >
>> > [2] https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541> <
>> > https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541>>
>> >
>> > Best,
>> > Paul Lam
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Shengkai Fang <fs...@gmail.com>.
Hi, Paul.
Thanks for your update. I have a few questions about the new design:
* How can we ship the json plan to the JobManager?
The current design only exposes an option about the URL of the json plan.
It seems the gateway is responsible to upload to an external stroage. Can
we reuse the PipelineOptions.JARS to ship to the remote filesystem?
* Classloading strategy
Currently, the Driver is in the sql-gateway package. It means the Driver is
not in the JM's classpath directly. Because the sql-gateway jar is now in
the opt directory rather than lib directory. It may need to add the
external dependencies as Python does[1]. BTW, I think it's better to move
the Driver into the flink-table-runtime package, which is much easier to
find(Sorry for the wrong opinion before).
* Option `$internal.sql-gateway.driver.sql-config` is string type
I think it's better to use Map type here
* PoC branch
Because this FLIP involves many modules, do you have a PoC branch to verify
it does work?
Best,
Shengkai
[1]
https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940
Paul Lam <pa...@gmail.com> 于2023年6月19日周一 14:09写道：
> Hi Shengkai,
>
> Sorry for my late reply. It took me some time to update the FLIP.
>
> In the latest FLIP design, SQL Driver is placed in flink-sql-gateway
> module. PTAL.
>
> The FLIP does not cover details about the K8s file distribution, but its
> general usage would
> be very much the same as YARN setups. We could make follow-up discussions
> in the jira
> tickets.
>
> Best,
> Paul Lam
>
> 2023年6月12日 15:29，Shengkai Fang <fs...@gmail.com> 写道：
>
>
> > If it’s the case, I’m good with introducing a new module and making SQL
> Driver
> > an internal class and accepts JSON plans only.
>
> I rethink this again and again. I think it's better to move the SqlDriver
> into the sql-gateway module because the sql client relies on the
> sql-gateway to submit the sql and the sql-gateway has the ability to
> generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
>
> * Upload configuration through command line parameter
>
> ExecNodeGraph only contains the job's information but it doesn't contain
> the checkpoint dir, checkpoint interval, execution mode and so on. So I
> think we should also upload the configuration.
>
> * KubernetesClusterDescripter and
> KubernetesApplicationClusterEntrypoint are responsible for the jar
> upload/download
>
> +1 for the change.
>
> Could you update the FLIP about the current discussion?
>
> Best,
> Shengkai
>
>
>
>
>
>
> Yang Wang <wa...@apache.org> 于2023年6月12日周一 11:41写道：
>
>> Sorry for the late reply. I am in favor of introducing such a built-in
>> resource localization mechanism
>> based on Flink FileSystem. Then FLINK-28915[1] could be the second step
>> which will download
>> the jars and dependencies to the JobManager/TaskManager local directory
>> before working.
>>
>> The first step could be done in another ticket in Flink. Or some external
>> Flink jobs management system
>> could also take care of this.
>>
>> [1]. https://issues.apache.org/jira/browse/FLINK-28915
>>
>> Best,
>> Yang
>>
>> Paul Lam <pa...@gmail.com> 于2023年6月9日周五 17:39写道：
>>
>> > Hi Mason,
>> >
>> > I get your point. I'm increasingly feeling the need to introduce a
>> > built-in
>> > file distribution mechanism for flink-kubernetes module, just like Spark
>> > does with `spark.kubernetes.file.upload.path` [1].
>> >
>> > I’m assuming the workflow is as follows:
>> >
>> > - KubernetesClusterDescripter uploads all local resources to a remote
>> > storage via Flink filesystem (skips if the resources are already
>> remote).
>> > - KubernetesApplicationClusterEntrypoint downloads the resources
>> > and put them in the classpath during startup.
>> >
>> > I wouldn't mind splitting it into another FLIP to ensure that
>> everything is
>> > done correctly.
>> >
>> > cc'ed @Yang to gather more opinions.
>> >
>> > [1]
>> >
>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月8日 12:15，Mason Chen <ma...@gmail.com> 写道：
>> >
>> > Hi Paul,
>> >
>> > Thanks for your response!
>> >
>> > I agree that utilizing SQL Drivers in Java applications is equally
>> > important
>> >
>> > as employing them in SQL Gateway. WRT init containers, I think most
>> > users use them just as a workaround. For example, wget a jar from the
>> > maven repo.
>> >
>> > We could implement the functionality in SQL Driver in a more graceful
>> > way and the flink-supported filesystem approach seems to be a
>> > good choice.
>> >
>> >
>> > My main point is: can we solve the problem with a design agnostic of SQL
>> > and Stream API? I mentioned a use case where this ability is useful for
>> > Java or Stream API applications. Maybe this is even a non-goal to your
>> FLIP
>> > since you are focusing on the driver entrypoint.
>> >
>> > Jark mentioned some optimizations:
>> >
>> > This allows SQLGateway to leverage some metadata caching and UDF JAR
>> > caching for better compiling performance.
>> >
>> > It would be great to see this even outside the SQLGateway (i.e. UDF JAR
>> > caching).
>> >
>> > Best,
>> > Mason
>> >
>> > On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fs...@gmail.com> wrote:
>> >
>> > Hi. Paul. Thanks for your update and the update makes me understand the
>> > design much better.
>> >
>> > But I still have some questions about the FLIP.
>> >
>> > For SQL Gateway, only DMLs need to be delegated to the SQL server
>> > Driver. I would think about the details and update the FLIP. Do you have
>> >
>> > some
>> >
>> > ideas already?
>> >
>> >
>> > If the applicaiton mode can not support library mode, I think we should
>> > only execute INSERT INTO and UPDATE/ DELETE statement in the application
>> > mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
>> > statements. The ANALYZE TABLE syntax need to register the statistic to
>> the
>> > catalog after job finishes and the CALL PROCEDURE statement doesn't
>> > generate the ExecNodeGraph.
>> >
>> > * Introduce storage via option `sql-gateway.application.storage-dir`
>> >
>> > If we can not support to submit the jars through web submission, +1 to
>> > introduce the options to upload the files. While I think the uploader
>> > should be responsible to remove the uploaded jars. Can we remove the
>> jars
>> > if the job is running or gateway exits?
>> >
>> > * JobID is not avaliable
>> >
>> > Can we use the returned rest client by ApplicationDeployer to query the
>> job
>> > id? I am concerned that users don't know which job is related to the
>> > submitted SQL.
>> >
>> > * Do we need to introduce a new module named flink-table-sql-runner?
>> >
>> > It seems we need to introduce a new module. Will the new module is
>> > available in the distribution package? I agree with Jark that we don't
>> need
>> > to introduce this for table-API users and these users have their main
>> > class. If we want to make users write the k8s operator more easily, I
>> think
>> > we should modify the k8s operator repo. If we don't need to support SQL
>> > files, can we make this jar only visible in the sql-gateway like we do
>> in
>> > the planner loader?[1]
>> >
>> > [1]
>> >
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
>> >
>> > Best,
>> > Shengkai
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Weihua Hu <hu...@gmail.com> 于2023年6月7日周三 10:52写道：
>> >
>> > Hi,
>> >
>> > Thanks for updating the FLIP.
>> >
>> > I have two cents on the distribution of SQLs and resources.
>> > 1. Should we support a common file distribution mechanism for k8s
>> > application mode?
>> > I have seen some issues and requirements on the mailing list.
>> > In our production environment, we implement the download command in the
>> > CliFrontend.
>> > And automatically add an init container to the POD for file
>> >
>> > downloading.
>> >
>> > The advantage of this
>> > is that we can use all Flink-supported file systems to store files.
>> >
>> > This need more discussion. I would appreciate hearing more opinions.
>> >
>> > 2. In this FLIP, we distribute files in two different ways in YARN and
>> > Kubernetes. Can we combine it in one way?
>> > If we don't want to implement a common file distribution for k8s
>> > application mode. Could we use the SQLDriver
>> > to download the files both in YARN and K8S? IMO, this can reduce the
>> >
>> > cost
>> >
>> > of code maintenance.
>> >
>> > Best,
>> > Weihua
>> >
>> >
>> > On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
>> >
>> > Hi Mason,
>> >
>> > Thanks for your input!
>> >
>> > +1 for init containers or a more generalized way of obtaining
>> >
>> > arbitrary
>> >
>> > files. File fetching isn't specific to just SQL--it also matters for
>> >
>> > Java
>> >
>> > applications if the user doesn't want to rebuild a Flink image and
>> >
>> > just
>> >
>> > wants to modify the user application fat jar.
>> >
>> >
>> > I agree that utilizing SQL Drivers in Java applications is equally
>> > important
>> > as employing them in SQL Gateway. WRT init containers, I think most
>> > users use them just as a workaround. For example, wget a jar from the
>> > maven repo.
>> >
>> > We could implement the functionality in SQL Driver in a more graceful
>> > way and the flink-supported filesystem approach seems to be a
>> > good choice.
>> >
>> > Also, what do you think about prefixing the config options with
>> > `sql-driver` instead of just `sql` to be more specific?
>> >
>> >
>> > LGTM, since SQL Driver is a public interface and the options are
>> > specific to it.
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
>> >
>> > Hi Paul,
>> >
>> > +1 for this feature and supporting SQL file + JSON plans. We get a
>> >
>> > lot
>> >
>> > of
>> >
>> > requests to just be able to submit a SQL file, but the JSON plan
>> > optimizations make sense.
>> >
>> > +1 for init containers or a more generalized way of obtaining
>> >
>> > arbitrary
>> >
>> > files. File fetching isn't specific to just SQL--it also matters for
>> >
>> > Java
>> >
>> > applications if the user doesn't want to rebuild a Flink image and
>> >
>> > just
>> >
>> > wants to modify the user application fat jar.
>> >
>> > Please note that we could reuse the checkpoint storage like S3/HDFS,
>> >
>> > which
>> >
>> > should
>> >
>> >
>> > be required to run Flink in production, so I guess that would be
>> >
>> > acceptable
>> >
>> > for most
>> >
>> >
>> > users. WDYT?
>> >
>> >
>> > If you do go this route, it would be nice to support writing these
>> >
>> > files
>> >
>> > to
>> >
>> > S3/HDFS via Flink. This makes access control and policy management
>> >
>> > simpler.
>> >
>> >
>> > Also, what do you think about prefixing the config options with
>> > `sql-driver` instead of just `sql` to be more specific?
>> >
>> > Best,
>> > Mason
>> >
>> > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
>> >
>> > <mailto:
>> >
>> > paullin3280@gmail.com>> wrote:
>> >
>> >
>> > Hi Jark,
>> >
>> > Thanks for your input! Please see my comments inline.
>> >
>> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>> > DataStream API also doesn't provide a default main class for users,
>> > why do we need to provide such one for SQL?
>> >
>> >
>> > Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>> >
>> > submitted
>> >
>> > via Flink CLI which actually could be DataStream/Table jobs.
>> >
>> > I think a default main class would be user-friendly which eliminates
>> >
>> > the
>> >
>> > need
>> > for users to write a main class as SQLRunner in Flink K8s operator
>> >
>> > [1].
>> >
>> >
>> > I thought the proposed SqlDriver was a dedicated main class
>> >
>> > accepting
>> >
>> > SQL files, is
>> >
>> > that correct?
>> >
>> >
>> > Both JSON plans and SQL files are accepted. SQL Gateway should use
>> >
>> > JSON
>> >
>> > plans,
>> > while CLI users may use either JSON plans or SQL files.
>> >
>> > Please see the updated FLIP[2] for more details.
>> >
>> > Personally, I prefer the way of init containers which doesn't
>> >
>> > depend
>> >
>> > on
>> >
>> > additional components.
>> > This can reduce the moving parts of a production environment.
>> > Depending on a distributed file system makes the testing, demo, and
>> >
>> > local
>> >
>> > setup harder than init containers.
>> >
>> >
>> > Please note that we could reuse the checkpoint storage like S3/HDFS,
>> >
>> > which
>> >
>> > should
>> > be required to run Flink in production, so I guess that would be
>> > acceptable for most
>> > users. WDYT?
>> >
>> > WRT testing, demo, and local setups, I think we could support the
>> >
>> > local
>> >
>> > filesystem
>> > scheme i.e. file://** as the state backends do. It works as long as
>> >
>> > SQL
>> >
>> > Gateway
>> > and JobManager(or SQL Driver) can access the resource directory
>> >
>> > (specified
>> >
>> > via
>> > `sql-gateway.application.storage-dir`).
>> >
>> > Thanks!
>> >
>> > [1]
>> >
>> >
>> >
>> >
>> >
>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
>> >
>> > [2]
>> >
>> >
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>> >
>> > [3]
>> >
>> >
>> >
>> >
>> >
>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
>> >
>> > Hi Paul,
>> >
>> > Thanks for your reply. I left my comments inline.
>> >
>> > As the FLIP said, it’s good to have a default main class for Flink
>> >
>> > SQLs,
>> >
>> > which allows users to submit Flink SQLs in the same way as
>> >
>> > DataStream
>> >
>> > jobs, or else users need to write their own main class.
>> >
>> >
>> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>> > DataStream API also doesn't provide a default main class for users,
>> > why do we need to provide such one for SQL?
>> >
>> > With the help of ExecNodeGraph, do we still need the serialized
>> > SessionState? If not, we could make SQL Driver accepts two
>> >
>> > serialized
>> >
>> > formats:
>> >
>> >
>> > No, ExecNodeGraph doesn't need to serialize SessionState. I thought
>> >
>> > the
>> >
>> > proposed SqlDriver was a dedicated main class accepting SQL files,
>> >
>> > is
>> >
>> > that correct?
>> > If true, we have to ship the SessionState for this case which is a
>> >
>> > large
>> >
>> > work.
>> > I think we just need a JsonPlanDriver which is a main class that
>> >
>> > accepts
>> >
>> > JsonPlan as the parameter.
>> >
>> >
>> > The common solutions I know is to use distributed file systems or
>> >
>> > use
>> >
>> > init containers to localize the resources.
>> >
>> >
>> > Personally, I prefer the way of init containers which doesn't
>> >
>> > depend
>> >
>> > on
>> >
>> > additional components.
>> > This can reduce the moving parts of a production environment.
>> > Depending on a distributed file system makes the testing, demo, and
>> >
>> > local
>> >
>> > setup harder than init containers.
>> >
>> > Best,
>> > Jark
>> >
>> >
>> >
>> >
>> > On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
>> >
>> > <mailto:
>> >
>> > paullin3280@gmail.com> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>> >
>> >
>> > The FLIP is in the early phase and some details are not included,
>> >
>> > but
>> >
>> > fortunately, we got lots of valuable ideas from the discussion.
>> >
>> > Thanks to everyone who joined the dissuasion!
>> > @Weihua @Shanmon @Shengkai @Biao @Jark
>> >
>> > This weekend I’m gonna revisit and update the FLIP, adding more
>> > details. Hopefully, we can further align our opinions.
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
>> >
>> > paullin3280@gmail.com>> 写道：
>> >
>> >
>> > Hi Jark,
>> >
>> > Thanks a lot for your input!
>> >
>> > If we decide to submit ExecNodeGraph instead of SQL file, is it
>> >
>> > still
>> >
>> > necessary to support SQL Driver?
>> >
>> >
>> > I think so. Apart from usage in SQL Gateway, SQL Driver could
>> >
>> > simplify
>> >
>> > Flink SQL execution with Flink CLI.
>> >
>> > As the FLIP said, it’s good to have a default main class for
>> >
>> > Flink
>> >
>> > SQLs,
>> >
>> > which allows users to submit Flink SQLs in the same way as
>> >
>> > DataStream
>> >
>> > jobs, or else users need to write their own main class.
>> >
>> > SQL Driver needs to serialize SessionState which is very
>> >
>> > challenging
>> >
>> > but not detailed covered in the FLIP.
>> >
>> >
>> > With the help of ExecNodeGraph, do we still need the serialized
>> > SessionState? If not, we could make SQL Driver accepts two
>> >
>> > serialized
>> >
>> > formats:
>> >
>> > - SQL files for user-facing public usage
>> > - ExecNodeGraph for internal usage
>> >
>> > It’s kind of similar to the relationship between job jars and
>> >
>> > jobgraphs.
>> >
>> >
>> > Regarding "K8S doesn't support shipping multiple jars", is that
>> >
>> > true?
>> >
>> > Is it
>> >
>> > possible to support it?
>> >
>> >
>> > Yes, K8s doesn’t distribute any files. It’s the users’
>> >
>> > responsibility
>> >
>> > to
>> >
>> > make
>> >
>> > sure the resources are accessible in the containers. The common
>> >
>> > solutions
>> >
>> > I know is to use distributed file systems or use init containers
>> >
>> > to
>> >
>> > localize the
>> >
>> > resources.
>> >
>> > Now I lean toward introducing a fs to do the distribution job.
>> >
>> > WDYT?
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
>> >
>> > imjark@gmail.com>
>> >
>> > <mailto:imjark@gmail.com <ma...@gmail.com>>
>> >
>> > <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
>> >
>> > imjark@gmail.com <ma...@gmail.com>>>>
>> >
>> > 写道：
>> >
>> >
>> > Hi Paul,
>> >
>> > Thanks for starting this discussion. I like the proposal! This
>> >
>> > is
>> >
>> > a
>> >
>> > frequently requested feature!
>> >
>> > I agree with Shengkai that ExecNodeGraph as the submission
>> >
>> > object
>> >
>> > is a
>> >
>> > better idea than SQL file. To be more specific, it should be
>> >
>> > JsonPlanGraph
>> >
>> > or CompiledPlan which is the serializable representation.
>> >
>> > CompiledPlan
>> >
>> > is a
>> >
>> > clear separation between compiling/optimization/validation and
>> >
>> > execution.
>> >
>> > This can keep the validation and metadata accessing still on the
>> >
>> > SQLGateway
>> >
>> > side. This allows SQLGateway to leverage some metadata caching
>> >
>> > and
>> >
>> > UDF
>> >
>> > JAR
>> >
>> > caching for better compiling performance.
>> >
>> > If we decide to submit ExecNodeGraph instead of SQL file, is it
>> >
>> > still
>> >
>> > necessary to support SQL Driver? Regarding non-interactive SQL
>> >
>> > jobs,
>> >
>> > users
>> >
>> > can use the Table API program for application mode. SQL Driver
>> >
>> > needs
>> >
>> > to
>> >
>> > serialize SessionState which is very challenging but not
>> >
>> > detailed
>> >
>> > covered
>> >
>> > in the FLIP.
>> >
>> > Regarding "K8S doesn't support shipping multiple jars", is that
>> >
>> > true?
>> >
>> > Is it
>> >
>> > possible to support it?
>> >
>> > Best,
>> > Jark
>> >
>> >
>> >
>> > On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
>> >
>> > <ma...@gmail.com> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> >
>> > paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>> >
>> >
>> > Hi Weihua,
>> >
>> > You’re right. Distributing the SQLs to the TMs is one of the
>> >
>> > challenging
>> >
>> > parts of this FLIP.
>> >
>> > Web submission is not enabled in application mode currently as
>> >
>> > you
>> >
>> > said,
>> >
>> > but it could be changed if we have good reasons.
>> >
>> > What do you think about introducing a distributed storage for
>> >
>> > SQL
>> >
>> > Gateway?
>> >
>> >
>> > We could make use of Flink file systems [1] to distribute the
>> >
>> > SQL
>> >
>> > Gateway
>> >
>> > generated resources, that should solve the problem at its root
>> >
>> > cause.
>> >
>> >
>> > Users could specify Flink-supported file systems to ship files.
>> >
>> > It’s
>> >
>> > only
>> >
>> > required when using SQL Gateway with K8s application mode.
>> >
>> > [1]
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>> >
>> > <
>> >
>> >
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>> >
>> > <
>> >
>> >
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>> >
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>> >
>> > <
>> >
>> >
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>> >
>> > <
>> >
>> >
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>> >
>> >
>> >
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
>> >
>> > huweihua.ckl@gmail.com> <mailto:
>> >
>> > huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
>> >
>> >
>> > Thanks Paul for your reply.
>> >
>> > SQLDriver looks good to me.
>> >
>> > 2. Do you mean a pass the SQL string a configuration or a
>> >
>> > program
>> >
>> > argument?
>> >
>> >
>> >
>> > I brought this up because we were unable to pass the SQL file
>> >
>> > to
>> >
>> > Flink
>> >
>> > using Kubernetes mode.
>> > For DataStream/Python users, they need to prepare their images
>> >
>> > for
>> >
>> > the
>> >
>> > jars
>> >
>> > and dependencies.
>> > But for SQL users, they can use a common image to run
>> >
>> > different
>> >
>> > SQL
>> >
>> > queries
>> >
>> > if there are no other udf requirements.
>> > It would be great if the SQL query and image were not bound.
>> >
>> > Using strings is a way to decouple these, but just as you
>> >
>> > mentioned,
>> >
>> > it's
>> >
>> > not easy to pass complex SQL.
>> >
>> > use web submission
>> >
>> > AFAIK, we can not use web submission in the Application mode.
>> >
>> > Please
>> >
>> > correct me if I'm wrong.
>> >
>> >
>> > Best,
>> > Weihua
>> >
>> >
>> > On Wed, May 31, 2023 at 9:37 PM Paul Lam <
>> >
>> > paullin3280@gmail.com
>> >
>> > <ma...@gmail.com>
>> >
>> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>> >
>> > wrote:
>> >
>> >
>> > Hi Biao,
>> >
>> > Thanks for your comments!
>> >
>> > 1. Scope: is this FLIP only targeted for non-interactive
>> >
>> > Flink
>> >
>> > SQL
>> >
>> > jobs
>> >
>> > in
>> >
>> > Application mode? More specifically, if we use SQL
>> >
>> > client/gateway
>> >
>> > to
>> >
>> > execute some interactive SQLs like a SELECT query, can we
>> >
>> > ask
>> >
>> > flink
>> >
>> > to
>> >
>> > use
>> >
>> > Application mode to execute those queries after this FLIP?
>> >
>> >
>> > Thanks for pointing it out. I think only DMLs would be
>> >
>> > executed
>> >
>> > via
>> >
>> > SQL
>> >
>> > Driver.
>> > I'll add the scope to the FLIP.
>> >
>> > 2. Deployment: I believe in YARN mode, the implementation is
>> >
>> > trivial as
>> >
>> > we
>> >
>> > can ship files via YARN's tool easily but for K8s, things
>> >
>> > can
>> >
>> > be
>> >
>> > more
>> >
>> > complicated as Shengkai said.
>> >
>> >
>> >
>> > Your input is very informative. I’m thinking about using web
>> >
>> > submission,
>> >
>> > but it requires exposing the JobManager port which could also
>> >
>> > be
>> >
>> > a
>> >
>> > problem
>> >
>> > on K8s.
>> >
>> > Another approach is to explicitly require a distributed
>> >
>> > storage
>> >
>> > to
>> >
>> > ship
>> >
>> > files,
>> > but we may need a new deployment executor for that.
>> >
>> > What do you think of these two approaches?
>> >
>> > 3. Serialization of SessionState: in SessionState, there are
>> >
>> > some
>> >
>> > unserializable fields
>> > like
>> >
>> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
>> >
>> > It
>> >
>> > may be worthwhile to add more details about the
>> >
>> > serialization
>> >
>> > part.
>> >
>> >
>> > I agree. That’s a missing part. But if we use ExecNodeGraph
>> >
>> > as
>> >
>> > Shengkai
>> >
>> > mentioned, do we eliminate the need for serialization of
>> >
>> > SessionState?
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
>> >
>> > biaogeng7@gmail.com> <mailto:
>> >
>> > biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
>> >
>> >
>> > Thanks Paul for the proposal!I believe it would be very
>> >
>> > useful
>> >
>> > for
>> >
>> > flink
>> >
>> > users.
>> > After reading the FLIP, I have some questions:
>> > 1. Scope: is this FLIP only targeted for non-interactive
>> >
>> > Flink
>> >
>> > SQL
>> >
>> > jobs
>> >
>> > in
>> >
>> > Application mode? More specifically, if we use SQL
>> >
>> > client/gateway
>> >
>> > to
>> >
>> > execute some interactive SQLs like a SELECT query, can we
>> >
>> > ask
>> >
>> > flink
>> >
>> > to
>> >
>> > use
>> >
>> > Application mode to execute those queries after this FLIP?
>> > 2. Deployment: I believe in YARN mode, the implementation is
>> >
>> > trivial as
>> >
>> > we
>> >
>> > can ship files via YARN's tool easily but for K8s, things
>> >
>> > can
>> >
>> > be
>> >
>> > more
>> >
>> > complicated as Shengkai said. I have implemented a simple
>> >
>> > POC
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>> >
>> > <
>> >
>> >
>> >
>> >
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>> >
>> > <
>> >
>> >
>> >
>> >
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>> >
>> >
>> >
>> >
>> > based on SQL client before(i.e. consider the SQL client
>> >
>> > which
>> >
>> > supports
>> >
>> > executing a SQL file as the SQL driver in this FLIP). One
>> >
>> > problem
>> >
>> > I
>> >
>> > have
>> >
>> > met is how do we ship SQL files ( or Job Graph) to the k8s
>> >
>> > side.
>> >
>> > Without
>> >
>> > such support, users have to modify the initContainer or
>> >
>> > rebuild
>> >
>> > a
>> >
>> > new
>> >
>> > K8s
>> >
>> > image every time to fetch the SQL file. Like the flink k8s
>> >
>> > operator,
>> >
>> > one
>> >
>> > workaround is to utilize the flink config(transforming the
>> >
>> > SQL
>> >
>> > file
>> >
>> > to
>> >
>> > a
>> >
>> > escaped string like Weihua mentioned) which will be
>> >
>> > converted
>> >
>> > to a
>> >
>> > ConfigMap but K8s has size limit of ConfigMaps(no larger
>> >
>> > than
>> >
>> > 1MB
>> >
>> > <
>> >
>> > https://kubernetes.io/docs/concepts/configuration/configmap/
>> >
>> > <
>> >
>> > https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> >
>> > https://kubernetes.io/docs/concepts/configuration/configmap/ <
>> >
>> > https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
>> >
>> > Not
>> >
>> > sure
>> >
>> > if we have better solutions.
>> > 3. Serialization of SessionState: in SessionState, there are
>> >
>> > some
>> >
>> > unserializable fields
>> > like
>> >
>> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
>> >
>> > It
>> >
>> > may be worthwhile to add more details about the
>> >
>> > serialization
>> >
>> > part.
>> >
>> >
>> > Best,
>> > Biao Geng
>> >
>> > Paul Lam <paullin3280@gmail.com <mailto:
>> >
>> > paullin3280@gmail.com
>> >
>> >
>> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>> >
>> > 于2023年5月31日周三 11:49写道：
>> >
>> >
>> > Hi Weihua,
>> >
>> > Thanks a lot for your input! Please see my comments inline.
>> >
>> > - Is SQLRunner the better name? We use this to run a SQL
>> >
>> > Job.
>> >
>> > (Not
>> >
>> > strong,
>> >
>> > the SQLDriver is fine for me)
>> >
>> >
>> > I’ve thought about SQL Runner but picked SQL Driver for the
>> >
>> > following
>> >
>> > reasons FYI:
>> >
>> > 1. I have a PythonDriver doing the same job for PyFlink [1]
>> > 2. Flink program's main class is sort of like Driver in
>> >
>> > JDBC
>> >
>> > which
>> >
>> > translates SQLs into
>> > databases specific languages.
>> >
>> > In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>> >
>> > - Could we run SQL jobs using SQL in strings? Otherwise,
>> >
>> > we
>> >
>> > need
>> >
>> > to
>> >
>> > prepare
>> >
>> > a SQL file in an image for Kubernetes application mode,
>> >
>> > which
>> >
>> > may
>> >
>> > be
>> >
>> > a
>> >
>> > bit
>> >
>> > cumbersome.
>> >
>> >
>> > Do you mean a pass the SQL string a configuration or a
>> >
>> > program
>> >
>> > argument?
>> >
>> >
>> > I thought it might be convenient for testing propose, but
>> >
>> > not
>> >
>> > recommended
>> >
>> > for production,
>> > cause Flink SQLs could be complicated and involves lots of
>> >
>> > characters
>> >
>> > that
>> >
>> > need to escape.
>> >
>> > WDYT?
>> >
>> > - I noticed that we don't specify the SQLDriver jar in the
>> >
>> > "run-application"
>> >
>> > command. Does that mean we need to perform automatic
>> >
>> > detection
>> >
>> > in
>> >
>> > Flink?
>> >
>> >
>> > Yes! It’s like running a PyFlink job with the following
>> >
>> > command:
>> >
>> >
>> > ```
>> > ./bin/flink run \
>> > --pyModule table.word_count \
>> > --pyFiles examples/python/table
>> > ```
>> >
>> > The CLI determines if it’s a SQL job, if yes apply the SQL
>> >
>> > Driver
>> >
>> > automatically.
>> >
>> >
>> > [1]
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>> >
>> > <
>> >
>> >
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>> >
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>> >
>> > <
>> >
>> >
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>> >
>> >
>> >
>> >
>> > Best,
>> > Paul Lam
>> >
>> > 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com
>> >
>> > <mailto:
>> >
>> > huweihua.ckl@gmail.com> <mailto:
>> >
>> > huweihua.ckl@gmail.com>> 写道：
>> >
>> >
>> > Thanks Paul for the proposal.
>> >
>> > +1 for this. It is valuable in improving ease of use.
>> >
>> > I have a few questions.
>> > - Is SQLRunner the better name? We use this to run a SQL
>> >
>> > Job.
>> >
>> > (Not
>> >
>> > strong,
>> >
>> > the SQLDriver is fine for me)
>> > - Could we run SQL jobs using SQL in strings? Otherwise,
>> >
>> > we
>> >
>> > need
>> >
>> > to
>> >
>> > prepare
>> >
>> > a SQL file in an image for Kubernetes application mode,
>> >
>> > which
>> >
>> > may
>> >
>> > be
>> >
>> > a
>> >
>> > bit
>> >
>> > cumbersome.
>> > - I noticed that we don't specify the SQLDriver jar in the
>> >
>> > "run-application"
>> >
>> > command. Does that mean we need to perform automatic
>> >
>> > detection
>> >
>> > in
>> >
>> > Flink?
>> >
>> >
>> >
>> > Best,
>> > Weihua
>> >
>> >
>> > On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>> >
>> > paullin3280@gmail.com
>> >
>> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>> >
>> > wrote:
>> >
>> >
>> > Hi team,
>> >
>> > I’d like to start a discussion about FLIP-316 [1], which
>> >
>> > introduces
>> >
>> > a
>> >
>> > SQL
>> >
>> > driver as the
>> > default main class for Flink SQL jobs.
>> >
>> > Currently, Flink SQL could be executed out of the box
>> >
>> > either
>> >
>> > via
>> >
>> > SQL
>> >
>> > Client/Gateway
>> > or embedded in a Flink Java/Python program.
>> >
>> > However, each one has its drawback:
>> >
>> > - SQL Client/Gateway doesn’t support the application
>> >
>> > deployment
>> >
>> > mode
>> >
>> > [2]
>> >
>> > - Flink Java/Python program requires extra work to write
>> >
>> > a
>> >
>> > non-SQL
>> >
>> > program
>> >
>> >
>> > Therefore, I propose adding a SQL driver to act as the
>> >
>> > default
>> >
>> > main
>> >
>> > class
>> >
>> > for SQL jobs.
>> > Please see the FLIP docs for details and feel free to
>> >
>> > comment.
>> >
>> > Thanks!
>> >
>> >
>> > [1]
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>> >
>> > <
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>> >
>> >
>> > [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>> > https://issues.apache.org/jira/browse/FLINK-26541>
>> >
>> > Best,
>> > Paul Lam
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Shengkai,
Sorry for my late reply. It took me some time to update the FLIP.
In the latest FLIP design, SQL Driver is placed in flink-sql-gateway module. PTAL.
The FLIP does not cover details about the K8s file distribution, but its general usage would
be very much the same as YARN setups. We could make follow-up discussions in the jira
tickets.
Best,
Paul Lam
> 2023年6月12日 15:29，Shengkai Fang <fs...@gmail.com> 写道：
>
>
> > If it’s the case, I’m good with introducing a new module and making SQL Driver
> > an internal class and accepts JSON plans only.
>
> I rethink this again and again. I think it's better to move the SqlDriver into the sql-gateway module because the sql client relies on the sql-gateway to submit the sql and the sql-gateway has the ability to generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
>
> * Upload configuration through command line parameter
>
> ExecNodeGraph only contains the job's information but it doesn't contain the checkpoint dir, checkpoint interval, execution mode and so on. So I think we should also upload the configuration.
>
> * KubernetesClusterDescripter and KubernetesApplicationClusterEntrypoint are responsible for the jar upload/download
>
> +1 for the change.
>
> Could you update the FLIP about the current discussion?
>
> Best,
> Shengkai
>
>
>
>
>
>
> Yang Wang <wangyang0918@apache.org <ma...@apache.org>> 于2023年6月12日周一 11:41写道：
> Sorry for the late reply. I am in favor of introducing such a built-in
> resource localization mechanism
> based on Flink FileSystem. Then FLINK-28915[1] could be the second step
> which will download
> the jars and dependencies to the JobManager/TaskManager local directory
> before working.
>
> The first step could be done in another ticket in Flink. Or some external
> Flink jobs management system
> could also take care of this.
>
> [1]. https://issues.apache.org/jira/browse/FLINK-28915 <https://issues.apache.org/jira/browse/FLINK-28915>
>
> Best,
> Yang
>
> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> 于2023年6月9日周五 17:39写道：
>
> > Hi Mason,
> >
> > I get your point. I'm increasingly feeling the need to introduce a
> > built-in
> > file distribution mechanism for flink-kubernetes module, just like Spark
> > does with `spark.kubernetes.file.upload.path` [1].
> >
> > I’m assuming the workflow is as follows:
> >
> > - KubernetesClusterDescripter uploads all local resources to a remote
> > storage via Flink filesystem (skips if the resources are already remote).
> > - KubernetesApplicationClusterEntrypoint downloads the resources
> > and put them in the classpath during startup.
> >
> > I wouldn't mind splitting it into another FLIP to ensure that everything is
> > done correctly.
> >
> > cc'ed @Yang to gather more opinions.
> >
> > [1]
> > https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management <https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management>
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月8日 12:15，Mason Chen <mas.chen6345@gmail.com <ma...@gmail.com>> 写道：
> >
> > Hi Paul,
> >
> > Thanks for your response!
> >
> > I agree that utilizing SQL Drivers in Java applications is equally
> > important
> >
> > as employing them in SQL Gateway. WRT init containers, I think most
> > users use them just as a workaround. For example, wget a jar from the
> > maven repo.
> >
> > We could implement the functionality in SQL Driver in a more graceful
> > way and the flink-supported filesystem approach seems to be a
> > good choice.
> >
> >
> > My main point is: can we solve the problem with a design agnostic of SQL
> > and Stream API? I mentioned a use case where this ability is useful for
> > Java or Stream API applications. Maybe this is even a non-goal to your FLIP
> > since you are focusing on the driver entrypoint.
> >
> > Jark mentioned some optimizations:
> >
> > This allows SQLGateway to leverage some metadata caching and UDF JAR
> > caching for better compiling performance.
> >
> > It would be great to see this even outside the SQLGateway (i.e. UDF JAR
> > caching).
> >
> > Best,
> > Mason
> >
> > On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskmine@gmail.com <ma...@gmail.com>> wrote:
> >
> > Hi. Paul. Thanks for your update and the update makes me understand the
> > design much better.
> >
> > But I still have some questions about the FLIP.
> >
> > For SQL Gateway, only DMLs need to be delegated to the SQL server
> > Driver. I would think about the details and update the FLIP. Do you have
> >
> > some
> >
> > ideas already?
> >
> >
> > If the applicaiton mode can not support library mode, I think we should
> > only execute INSERT INTO and UPDATE/ DELETE statement in the application
> > mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> > statements. The ANALYZE TABLE syntax need to register the statistic to the
> > catalog after job finishes and the CALL PROCEDURE statement doesn't
> > generate the ExecNodeGraph.
> >
> > * Introduce storage via option `sql-gateway.application.storage-dir`
> >
> > If we can not support to submit the jars through web submission, +1 to
> > introduce the options to upload the files. While I think the uploader
> > should be responsible to remove the uploaded jars. Can we remove the jars
> > if the job is running or gateway exits?
> >
> > * JobID is not avaliable
> >
> > Can we use the returned rest client by ApplicationDeployer to query the job
> > id? I am concerned that users don't know which job is related to the
> > submitted SQL.
> >
> > * Do we need to introduce a new module named flink-table-sql-runner?
> >
> > It seems we need to introduce a new module. Will the new module is
> > available in the distribution package? I agree with Jark that we don't need
> > to introduce this for table-API users and these users have their main
> > class. If we want to make users write the k8s operator more easily, I think
> > we should modify the k8s operator repo. If we don't need to support SQL
> > files, can we make this jar only visible in the sql-gateway like we do in
> > the planner loader?[1]
> >
> > [1]
> >
> >
> > https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 <https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95>
> >
> > Best,
> > Shengkai
> >
> >
> >
> >
> >
> >
> >
> >
> > Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com>> 于2023年6月7日周三 10:52写道：
> >
> > Hi,
> >
> > Thanks for updating the FLIP.
> >
> > I have two cents on the distribution of SQLs and resources.
> > 1. Should we support a common file distribution mechanism for k8s
> > application mode?
> > I have seen some issues and requirements on the mailing list.
> > In our production environment, we implement the download command in the
> > CliFrontend.
> > And automatically add an init container to the POD for file
> >
> > downloading.
> >
> > The advantage of this
> > is that we can use all Flink-supported file systems to store files.
> >
> > This need more discussion. I would appreciate hearing more opinions.
> >
> > 2. In this FLIP, we distribute files in two different ways in YARN and
> > Kubernetes. Can we combine it in one way?
> > If we don't want to implement a common file distribution for k8s
> > application mode. Could we use the SQLDriver
> > to download the files both in YARN and K8S? IMO, this can reduce the
> >
> > cost
> >
> > of code maintenance.
> >
> > Best,
> > Weihua
> >
> >
> > On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
> >
> > Hi Mason,
> >
> > Thanks for your input!
> >
> > +1 for init containers or a more generalized way of obtaining
> >
> > arbitrary
> >
> > files. File fetching isn't specific to just SQL--it also matters for
> >
> > Java
> >
> > applications if the user doesn't want to rebuild a Flink image and
> >
> > just
> >
> > wants to modify the user application fat jar.
> >
> >
> > I agree that utilizing SQL Drivers in Java applications is equally
> > important
> > as employing them in SQL Gateway. WRT init containers, I think most
> > users use them just as a workaround. For example, wget a jar from the
> > maven repo.
> >
> > We could implement the functionality in SQL Driver in a more graceful
> > way and the flink-supported filesystem approach seems to be a
> > good choice.
> >
> > Also, what do you think about prefixing the config options with
> > `sql-driver` instead of just `sql` to be more specific?
> >
> >
> > LGTM, since SQL Driver is a public interface and the options are
> > specific to it.
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月6日 06:30，Mason Chen <mas.chen6345@gmail.com <ma...@gmail.com>> 写道：
> >
> > Hi Paul,
> >
> > +1 for this feature and supporting SQL file + JSON plans. We get a
> >
> > lot
> >
> > of
> >
> > requests to just be able to submit a SQL file, but the JSON plan
> > optimizations make sense.
> >
> > +1 for init containers or a more generalized way of obtaining
> >
> > arbitrary
> >
> > files. File fetching isn't specific to just SQL--it also matters for
> >
> > Java
> >
> > applications if the user doesn't want to rebuild a Flink image and
> >
> > just
> >
> > wants to modify the user application fat jar.
> >
> > Please note that we could reuse the checkpoint storage like S3/HDFS,
> >
> > which
> >
> > should
> >
> >
> > be required to run Flink in production, so I guess that would be
> >
> > acceptable
> >
> > for most
> >
> >
> > users. WDYT?
> >
> >
> > If you do go this route, it would be nice to support writing these
> >
> > files
> >
> > to
> >
> > S3/HDFS via Flink. This makes access control and policy management
> >
> > simpler.
> >
> >
> > Also, what do you think about prefixing the config options with
> > `sql-driver` instead of just `sql` to be more specific?
> >
> > Best,
> > Mason
> >
> > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> >
> > <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com>>> wrote:
> >
> >
> > Hi Jark,
> >
> > Thanks for your input! Please see my comments inline.
> >
> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > DataStream API also doesn't provide a default main class for users,
> > why do we need to provide such one for SQL?
> >
> >
> > Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> >
> > submitted
> >
> > via Flink CLI which actually could be DataStream/Table jobs.
> >
> > I think a default main class would be user-friendly which eliminates
> >
> > the
> >
> > need
> > for users to write a main class as SQLRunner in Flink K8s operator
> >
> > [1].
> >
> >
> > I thought the proposed SqlDriver was a dedicated main class
> >
> > accepting
> >
> > SQL files, is
> >
> > that correct?
> >
> >
> > Both JSON plans and SQL files are accepted. SQL Gateway should use
> >
> > JSON
> >
> > plans,
> > while CLI users may use either JSON plans or SQL files.
> >
> > Please see the updated FLIP[2] for more details.
> >
> > Personally, I prefer the way of init containers which doesn't
> >
> > depend
> >
> > on
> >
> > additional components.
> > This can reduce the moving parts of a production environment.
> > Depending on a distributed file system makes the testing, demo, and
> >
> > local
> >
> > setup harder than init containers.
> >
> >
> > Please note that we could reuse the checkpoint storage like S3/HDFS,
> >
> > which
> >
> > should
> > be required to run Flink in production, so I guess that would be
> > acceptable for most
> > users. WDYT?
> >
> > WRT testing, demo, and local setups, I think we could support the
> >
> > local
> >
> > filesystem
> > scheme i.e. file://** as the state backends do. It works as long as
> >
> > SQL
> >
> > Gateway
> > and JobManager(or SQL Driver) can access the resource directory
> >
> > (specified
> >
> > via
> > `sql-gateway.application.storage-dir`).
> >
> > Thanks!
> >
> > [1]
> >
> >
> >
> >
> > https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java <https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java>
> >
> > [2]
> >
> >
> >
> >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
> >
> > [3]
> >
> >
> >
> >
> > https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 <https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161>
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月3日 12:21，Jark Wu <imjark@gmail.com <ma...@gmail.com>> 写道：
> >
> > Hi Paul,
> >
> > Thanks for your reply. I left my comments inline.
> >
> > As the FLIP said, it’s good to have a default main class for Flink
> >
> > SQLs,
> >
> > which allows users to submit Flink SQLs in the same way as
> >
> > DataStream
> >
> > jobs, or else users need to write their own main class.
> >
> >
> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > DataStream API also doesn't provide a default main class for users,
> > why do we need to provide such one for SQL?
> >
> > With the help of ExecNodeGraph, do we still need the serialized
> > SessionState? If not, we could make SQL Driver accepts two
> >
> > serialized
> >
> > formats:
> >
> >
> > No, ExecNodeGraph doesn't need to serialize SessionState. I thought
> >
> > the
> >
> > proposed SqlDriver was a dedicated main class accepting SQL files,
> >
> > is
> >
> > that correct?
> > If true, we have to ship the SessionState for this case which is a
> >
> > large
> >
> > work.
> > I think we just need a JsonPlanDriver which is a main class that
> >
> > accepts
> >
> > JsonPlan as the parameter.
> >
> >
> > The common solutions I know is to use distributed file systems or
> >
> > use
> >
> > init containers to localize the resources.
> >
> >
> > Personally, I prefer the way of init containers which doesn't
> >
> > depend
> >
> > on
> >
> > additional components.
> > This can reduce the moving parts of a production environment.
> > Depending on a distributed file system makes the testing, demo, and
> >
> > local
> >
> > setup harder than init containers.
> >
> > Best,
> > Jark
> >
> >
> >
> >
> > On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> >
> > <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
> >
> >
> > The FLIP is in the early phase and some details are not included,
> >
> > but
> >
> > fortunately, we got lots of valuable ideas from the discussion.
> >
> > Thanks to everyone who joined the dissuasion!
> > @Weihua @Shanmon @Shengkai @Biao @Jark
> >
> > This weekend I’m gonna revisit and update the FLIP, adding more
> > details. Hopefully, we can further align our opinions.
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com>>> 写道：
> >
> >
> > Hi Jark,
> >
> > Thanks a lot for your input!
> >
> > If we decide to submit ExecNodeGraph instead of SQL file, is it
> >
> > still
> >
> > necessary to support SQL Driver?
> >
> >
> > I think so. Apart from usage in SQL Gateway, SQL Driver could
> >
> > simplify
> >
> > Flink SQL execution with Flink CLI.
> >
> > As the FLIP said, it’s good to have a default main class for
> >
> > Flink
> >
> > SQLs,
> >
> > which allows users to submit Flink SQLs in the same way as
> >
> > DataStream
> >
> > jobs, or else users need to write their own main class.
> >
> > SQL Driver needs to serialize SessionState which is very
> >
> > challenging
> >
> > but not detailed covered in the FLIP.
> >
> >
> > With the help of ExecNodeGraph, do we still need the serialized
> > SessionState? If not, we could make SQL Driver accepts two
> >
> > serialized
> >
> > formats:
> >
> > - SQL files for user-facing public usage
> > - ExecNodeGraph for internal usage
> >
> > It’s kind of similar to the relationship between job jars and
> >
> > jobgraphs.
> >
> >
> > Regarding "K8S doesn't support shipping multiple jars", is that
> >
> > true?
> >
> > Is it
> >
> > possible to support it?
> >
> >
> > Yes, K8s doesn’t distribute any files. It’s the users’
> >
> > responsibility
> >
> > to
> >
> > make
> >
> > sure the resources are accessible in the containers. The common
> >
> > solutions
> >
> > I know is to use distributed file systems or use init containers
> >
> > to
> >
> > localize the
> >
> > resources.
> >
> > Now I lean toward introducing a fs to do the distribution job.
> >
> > WDYT?
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com> <mailto:
> >
> > imjark@gmail.com <ma...@gmail.com>>
> >
> > <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>
> >
> > <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>> <mailto:
> >
> > imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>>>
> >
> > 写道：
> >
> >
> > Hi Paul,
> >
> > Thanks for starting this discussion. I like the proposal! This
> >
> > is
> >
> > a
> >
> > frequently requested feature!
> >
> > I agree with Shengkai that ExecNodeGraph as the submission
> >
> > object
> >
> > is a
> >
> > better idea than SQL file. To be more specific, it should be
> >
> > JsonPlanGraph
> >
> > or CompiledPlan which is the serializable representation.
> >
> > CompiledPlan
> >
> > is a
> >
> > clear separation between compiling/optimization/validation and
> >
> > execution.
> >
> > This can keep the validation and metadata accessing still on the
> >
> > SQLGateway
> >
> > side. This allows SQLGateway to leverage some metadata caching
> >
> > and
> >
> > UDF
> >
> > JAR
> >
> > caching for better compiling performance.
> >
> > If we decide to submit ExecNodeGraph instead of SQL file, is it
> >
> > still
> >
> > necessary to support SQL Driver? Regarding non-interactive SQL
> >
> > jobs,
> >
> > users
> >
> > can use the Table API program for application mode. SQL Driver
> >
> > needs
> >
> > to
> >
> > serialize SessionState which is very challenging but not
> >
> > detailed
> >
> > covered
> >
> > in the FLIP.
> >
> > Regarding "K8S doesn't support shipping multiple jars", is that
> >
> > true?
> >
> > Is it
> >
> > possible to support it?
> >
> > Best,
> > Jark
> >
> >
> >
> > On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>> wrote:
> >
> >
> > Hi Weihua,
> >
> > You’re right. Distributing the SQLs to the TMs is one of the
> >
> > challenging
> >
> > parts of this FLIP.
> >
> > Web submission is not enabled in application mode currently as
> >
> > you
> >
> > said,
> >
> > but it could be changed if we have good reasons.
> >
> > What do you think about introducing a distributed storage for
> >
> > SQL
> >
> > Gateway?
> >
> >
> > We could make use of Flink file systems [1] to distribute the
> >
> > SQL
> >
> > Gateway
> >
> > generated resources, that should solve the problem at its root
> >
> > cause.
> >
> >
> > Users could specify Flink-supported file systems to ship files.
> >
> > It’s
> >
> > only
> >
> > required when using SQL Gateway with K8s application mode.
> >
> > [1]
> >
> >
> >
> >
> >
> >
> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
> >
> > <
> >
> >
> >
> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
> >
> >
> > <
> >
> >
> >
> >
> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
> >
> > <
> >
> >
> >
> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
> >
> >
> >
> > <
> >
> >
> >
> >
> >
> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
> >
> > <
> >
> >
> >
> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
> >
> >
> > <
> >
> >
> >
> >
> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
> >
> > <
> >
> >
> >
> > https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
> >
> >
> >
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
> >
> > huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
> >
> > huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>>>> 写道：
> >
> >
> > Thanks Paul for your reply.
> >
> > SQLDriver looks good to me.
> >
> > 2. Do you mean a pass the SQL string a configuration or a
> >
> > program
> >
> > argument?
> >
> >
> >
> > I brought this up because we were unable to pass the SQL file
> >
> > to
> >
> > Flink
> >
> > using Kubernetes mode.
> > For DataStream/Python users, they need to prepare their images
> >
> > for
> >
> > the
> >
> > jars
> >
> > and dependencies.
> > But for SQL users, they can use a common image to run
> >
> > different
> >
> > SQL
> >
> > queries
> >
> > if there are no other udf requirements.
> > It would be great if the SQL query and image were not bound.
> >
> > Using strings is a way to decouple these, but just as you
> >
> > mentioned,
> >
> > it's
> >
> > not easy to pass complex SQL.
> >
> > use web submission
> >
> > AFAIK, we can not use web submission in the Application mode.
> >
> > Please
> >
> > correct me if I'm wrong.
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Wed, May 31, 2023 at 9:37 PM Paul Lam <
> >
> > paullin3280@gmail.com <ma...@gmail.com>
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
> >
> > wrote:
> >
> >
> > Hi Biao,
> >
> > Thanks for your comments!
> >
> > 1. Scope: is this FLIP only targeted for non-interactive
> >
> > Flink
> >
> > SQL
> >
> > jobs
> >
> > in
> >
> > Application mode? More specifically, if we use SQL
> >
> > client/gateway
> >
> > to
> >
> > execute some interactive SQLs like a SELECT query, can we
> >
> > ask
> >
> > flink
> >
> > to
> >
> > use
> >
> > Application mode to execute those queries after this FLIP?
> >
> >
> > Thanks for pointing it out. I think only DMLs would be
> >
> > executed
> >
> > via
> >
> > SQL
> >
> > Driver.
> > I'll add the scope to the FLIP.
> >
> > 2. Deployment: I believe in YARN mode, the implementation is
> >
> > trivial as
> >
> > we
> >
> > can ship files via YARN's tool easily but for K8s, things
> >
> > can
> >
> > be
> >
> > more
> >
> > complicated as Shengkai said.
> >
> >
> >
> > Your input is very informative. I’m thinking about using web
> >
> > submission,
> >
> > but it requires exposing the JobManager port which could also
> >
> > be
> >
> > a
> >
> > problem
> >
> > on K8s.
> >
> > Another approach is to explicitly require a distributed
> >
> > storage
> >
> > to
> >
> > ship
> >
> > files,
> > but we may need a new deployment executor for that.
> >
> > What do you think of these two approaches?
> >
> > 3. Serialization of SessionState: in SessionState, there are
> >
> > some
> >
> > unserializable fields
> > like
> >
> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >
> > It
> >
> > may be worthwhile to add more details about the
> >
> > serialization
> >
> > part.
> >
> >
> > I agree. That’s a missing part. But if we use ExecNodeGraph
> >
> > as
> >
> > Shengkai
> >
> > mentioned, do we eliminate the need for serialization of
> >
> > SessionState?
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <ma...@gmail.com> <mailto:
> >
> > biaogeng7@gmail.com <ma...@gmail.com>> <mailto:
> >
> > biaogeng7@gmail.com <ma...@gmail.com> <mailto:biaogeng7@gmail.com <ma...@gmail.com>>>> 写道：
> >
> >
> > Thanks Paul for the proposal!I believe it would be very
> >
> > useful
> >
> > for
> >
> > flink
> >
> > users.
> > After reading the FLIP, I have some questions:
> > 1. Scope: is this FLIP only targeted for non-interactive
> >
> > Flink
> >
> > SQL
> >
> > jobs
> >
> > in
> >
> > Application mode? More specifically, if we use SQL
> >
> > client/gateway
> >
> > to
> >
> > execute some interactive SQLs like a SELECT query, can we
> >
> > ask
> >
> > flink
> >
> > to
> >
> > use
> >
> > Application mode to execute those queries after this FLIP?
> > 2. Deployment: I believe in YARN mode, the implementation is
> >
> > trivial as
> >
> > we
> >
> > can ship files via YARN's tool easily but for K8s, things
> >
> > can
> >
> > be
> >
> > more
> >
> > complicated as Shengkai said. I have implemented a simple
> >
> > POC
> >
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
> >
> > <
> >
> >
> >
> > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
> >
> >
> > <
> >
> >
> >
> >
> > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
> >
> > <
> >
> >
> >
> > https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
> >
> >
> >
> >
> > based on SQL client before(i.e. consider the SQL client
> >
> > which
> >
> > supports
> >
> > executing a SQL file as the SQL driver in this FLIP). One
> >
> > problem
> >
> > I
> >
> > have
> >
> > met is how do we ship SQL files ( or Job Graph) to the k8s
> >
> > side.
> >
> > Without
> >
> > such support, users have to modify the initContainer or
> >
> > rebuild
> >
> > a
> >
> > new
> >
> > K8s
> >
> > image every time to fetch the SQL file. Like the flink k8s
> >
> > operator,
> >
> > one
> >
> > workaround is to utilize the flink config(transforming the
> >
> > SQL
> >
> > file
> >
> > to
> >
> > a
> >
> > escaped string like Weihua mentioned) which will be
> >
> > converted
> >
> > to a
> >
> > ConfigMap but K8s has size limit of ConfigMaps(no larger
> >
> > than
> >
> > 1MB
> >
> > <
> >
> > https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>
> >
> > <
> >
> > https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>> <
> >
> > https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
> >
> > https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>>>>).
> >
> > Not
> >
> > sure
> >
> > if we have better solutions.
> > 3. Serialization of SessionState: in SessionState, there are
> >
> > some
> >
> > unserializable fields
> > like
> >
> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >
> > It
> >
> > may be worthwhile to add more details about the
> >
> > serialization
> >
> > part.
> >
> >
> > Best,
> > Biao Geng
> >
> > Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com>
> >
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
> >
> > 于2023年5月31日周三 11:49写道：
> >
> >
> > Hi Weihua,
> >
> > Thanks a lot for your input! Please see my comments inline.
> >
> > - Is SQLRunner the better name? We use this to run a SQL
> >
> > Job.
> >
> > (Not
> >
> > strong,
> >
> > the SQLDriver is fine for me)
> >
> >
> > I’ve thought about SQL Runner but picked SQL Driver for the
> >
> > following
> >
> > reasons FYI:
> >
> > 1. I have a PythonDriver doing the same job for PyFlink [1]
> > 2. Flink program's main class is sort of like Driver in
> >
> > JDBC
> >
> > which
> >
> > translates SQLs into
> > databases specific languages.
> >
> > In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >
> > - Could we run SQL jobs using SQL in strings? Otherwise,
> >
> > we
> >
> > need
> >
> > to
> >
> > prepare
> >
> > a SQL file in an image for Kubernetes application mode,
> >
> > which
> >
> > may
> >
> > be
> >
> > a
> >
> > bit
> >
> > cumbersome.
> >
> >
> > Do you mean a pass the SQL string a configuration or a
> >
> > program
> >
> > argument?
> >
> >
> > I thought it might be convenient for testing propose, but
> >
> > not
> >
> > recommended
> >
> > for production,
> > cause Flink SQLs could be complicated and involves lots of
> >
> > characters
> >
> > that
> >
> > need to escape.
> >
> > WDYT?
> >
> > - I noticed that we don't specify the SQLDriver jar in the
> >
> > "run-application"
> >
> > command. Does that mean we need to perform automatic
> >
> > detection
> >
> > in
> >
> > Flink?
> >
> >
> > Yes! It’s like running a PyFlink job with the following
> >
> > command:
> >
> >
> > ```
> > ./bin/flink run \
> > --pyModule table.word_count \
> > --pyFiles examples/python/table
> > ```
> >
> > The CLI determines if it’s a SQL job, if yes apply the SQL
> >
> > Driver
> >
> > automatically.
> >
> >
> > [1]
> >
> >
> >
> >
> >
> >
> >
> >
> > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
> >
> > <
> >
> >
> >
> > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
> >
> >
> > <
> >
> >
> >
> >
> > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
> >
> > <
> >
> >
> >
> > https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
> >
> >
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com>
> >
> > <mailto:
> >
> > huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
> >
> > huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
> >
> >
> > Thanks Paul for the proposal.
> >
> > +1 for this. It is valuable in improving ease of use.
> >
> > I have a few questions.
> > - Is SQLRunner the better name? We use this to run a SQL
> >
> > Job.
> >
> > (Not
> >
> > strong,
> >
> > the SQLDriver is fine for me)
> > - Could we run SQL jobs using SQL in strings? Otherwise,
> >
> > we
> >
> > need
> >
> > to
> >
> > prepare
> >
> > a SQL file in an image for Kubernetes application mode,
> >
> > which
> >
> > may
> >
> > be
> >
> > a
> >
> > bit
> >
> > cumbersome.
> > - I noticed that we don't specify the SQLDriver jar in the
> >
> > "run-application"
> >
> > command. Does that mean we need to perform automatic
> >
> > detection
> >
> > in
> >
> > Flink?
> >
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> >
> > paullin3280@gmail.com <ma...@gmail.com>
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
> >
> > wrote:
> >
> >
> > Hi team,
> >
> > I’d like to start a discussion about FLIP-316 [1], which
> >
> > introduces
> >
> > a
> >
> > SQL
> >
> > driver as the
> > default main class for Flink SQL jobs.
> >
> > Currently, Flink SQL could be executed out of the box
> >
> > either
> >
> > via
> >
> > SQL
> >
> > Client/Gateway
> > or embedded in a Flink Java/Python program.
> >
> > However, each one has its drawback:
> >
> > - SQL Client/Gateway doesn’t support the application
> >
> > deployment
> >
> > mode
> >
> > [2]
> >
> > - Flink Java/Python program requires extra work to write
> >
> > a
> >
> > non-SQL
> >
> > program
> >
> >
> > Therefore, I propose adding a SQL driver to act as the
> >
> > default
> >
> > main
> >
> > class
> >
> > for SQL jobs.
> > Please see the FLIP docs for details and feel free to
> >
> > comment.
> >
> > Thanks!
> >
> >
> > [1]
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver>
> >
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
> >
> >
> > [2] https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541> <
> > https://issues.apache.org/jira/browse/FLINK-26541 <https://issues.apache.org/jira/browse/FLINK-26541>>
> >
> > Best,
> > Paul Lam
> >
> >
> >
> >
> >
> >
> >
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Shengkai Fang <fs...@gmail.com>.
> If it’s the case, I’m good with introducing a new module and making SQL
Driver
> an internal class and accepts JSON plans only.
I rethink this again and again. I think it's better to move the SqlDriver
into the sql-gateway module because the sql client relies on the
sql-gateway to submit the sql and the sql-gateway has the ability to
generate the ExecNodeGraph now. +1 to support accepting JSON plans only.
* Upload configuration through command line parameter
ExecNodeGraph only contains the job's information but it doesn't contain
the checkpoint dir, checkpoint interval, execution mode and so on. So I
think we should also upload the configuration.
* KubernetesClusterDescripter and
KubernetesApplicationClusterEntrypoint are responsible for the jar
upload/download
+1 for the change.
Could you update the FLIP about the current discussion?
Best,
Shengkai
Yang Wang <wa...@apache.org> 于2023年6月12日周一 11:41写道：
> Sorry for the late reply. I am in favor of introducing such a built-in
> resource localization mechanism
> based on Flink FileSystem. Then FLINK-28915[1] could be the second step
> which will download
> the jars and dependencies to the JobManager/TaskManager local directory
> before working.
>
> The first step could be done in another ticket in Flink. Or some external
> Flink jobs management system
> could also take care of this.
>
> [1]. https://issues.apache.org/jira/browse/FLINK-28915
>
> Best,
> Yang
>
> Paul Lam <pa...@gmail.com> 于2023年6月9日周五 17:39写道：
>
> > Hi Mason,
> >
> > I get your point. I'm increasingly feeling the need to introduce a
> > built-in
> > file distribution mechanism for flink-kubernetes module, just like Spark
> > does with `spark.kubernetes.file.upload.path` [1].
> >
> > I’m assuming the workflow is as follows:
> >
> > - KubernetesClusterDescripter uploads all local resources to a remote
> > storage via Flink filesystem (skips if the resources are already
> remote).
> > - KubernetesApplicationClusterEntrypoint downloads the resources
> > and put them in the classpath during startup.
> >
> > I wouldn't mind splitting it into another FLIP to ensure that everything
> is
> > done correctly.
> >
> > cc'ed @Yang to gather more opinions.
> >
> > [1]
> >
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月8日 12:15，Mason Chen <ma...@gmail.com> 写道：
> >
> > Hi Paul,
> >
> > Thanks for your response!
> >
> > I agree that utilizing SQL Drivers in Java applications is equally
> > important
> >
> > as employing them in SQL Gateway. WRT init containers, I think most
> > users use them just as a workaround. For example, wget a jar from the
> > maven repo.
> >
> > We could implement the functionality in SQL Driver in a more graceful
> > way and the flink-supported filesystem approach seems to be a
> > good choice.
> >
> >
> > My main point is: can we solve the problem with a design agnostic of SQL
> > and Stream API? I mentioned a use case where this ability is useful for
> > Java or Stream API applications. Maybe this is even a non-goal to your
> FLIP
> > since you are focusing on the driver entrypoint.
> >
> > Jark mentioned some optimizations:
> >
> > This allows SQLGateway to leverage some metadata caching and UDF JAR
> > caching for better compiling performance.
> >
> > It would be great to see this even outside the SQLGateway (i.e. UDF JAR
> > caching).
> >
> > Best,
> > Mason
> >
> > On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fs...@gmail.com> wrote:
> >
> > Hi. Paul. Thanks for your update and the update makes me understand the
> > design much better.
> >
> > But I still have some questions about the FLIP.
> >
> > For SQL Gateway, only DMLs need to be delegated to the SQL server
> > Driver. I would think about the details and update the FLIP. Do you have
> >
> > some
> >
> > ideas already?
> >
> >
> > If the applicaiton mode can not support library mode, I think we should
> > only execute INSERT INTO and UPDATE/ DELETE statement in the application
> > mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> > statements. The ANALYZE TABLE syntax need to register the statistic to
> the
> > catalog after job finishes and the CALL PROCEDURE statement doesn't
> > generate the ExecNodeGraph.
> >
> > * Introduce storage via option `sql-gateway.application.storage-dir`
> >
> > If we can not support to submit the jars through web submission, +1 to
> > introduce the options to upload the files. While I think the uploader
> > should be responsible to remove the uploaded jars. Can we remove the jars
> > if the job is running or gateway exits?
> >
> > * JobID is not avaliable
> >
> > Can we use the returned rest client by ApplicationDeployer to query the
> job
> > id? I am concerned that users don't know which job is related to the
> > submitted SQL.
> >
> > * Do we need to introduce a new module named flink-table-sql-runner?
> >
> > It seems we need to introduce a new module. Will the new module is
> > available in the distribution package? I agree with Jark that we don't
> need
> > to introduce this for table-API users and these users have their main
> > class. If we want to make users write the k8s operator more easily, I
> think
> > we should modify the k8s operator repo. If we don't need to support SQL
> > files, can we make this jar only visible in the sql-gateway like we do in
> > the planner loader?[1]
> >
> > [1]
> >
> >
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
> >
> > Best,
> > Shengkai
> >
> >
> >
> >
> >
> >
> >
> >
> > Weihua Hu <hu...@gmail.com> 于2023年6月7日周三 10:52写道：
> >
> > Hi,
> >
> > Thanks for updating the FLIP.
> >
> > I have two cents on the distribution of SQLs and resources.
> > 1. Should we support a common file distribution mechanism for k8s
> > application mode?
> > I have seen some issues and requirements on the mailing list.
> > In our production environment, we implement the download command in the
> > CliFrontend.
> > And automatically add an init container to the POD for file
> >
> > downloading.
> >
> > The advantage of this
> > is that we can use all Flink-supported file systems to store files.
> >
> > This need more discussion. I would appreciate hearing more opinions.
> >
> > 2. In this FLIP, we distribute files in two different ways in YARN and
> > Kubernetes. Can we combine it in one way?
> > If we don't want to implement a common file distribution for k8s
> > application mode. Could we use the SQLDriver
> > to download the files both in YARN and K8S? IMO, this can reduce the
> >
> > cost
> >
> > of code maintenance.
> >
> > Best,
> > Weihua
> >
> >
> > On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
> >
> > Hi Mason,
> >
> > Thanks for your input!
> >
> > +1 for init containers or a more generalized way of obtaining
> >
> > arbitrary
> >
> > files. File fetching isn't specific to just SQL--it also matters for
> >
> > Java
> >
> > applications if the user doesn't want to rebuild a Flink image and
> >
> > just
> >
> > wants to modify the user application fat jar.
> >
> >
> > I agree that utilizing SQL Drivers in Java applications is equally
> > important
> > as employing them in SQL Gateway. WRT init containers, I think most
> > users use them just as a workaround. For example, wget a jar from the
> > maven repo.
> >
> > We could implement the functionality in SQL Driver in a more graceful
> > way and the flink-supported filesystem approach seems to be a
> > good choice.
> >
> > Also, what do you think about prefixing the config options with
> > `sql-driver` instead of just `sql` to be more specific?
> >
> >
> > LGTM, since SQL Driver is a public interface and the options are
> > specific to it.
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
> >
> > Hi Paul,
> >
> > +1 for this feature and supporting SQL file + JSON plans. We get a
> >
> > lot
> >
> > of
> >
> > requests to just be able to submit a SQL file, but the JSON plan
> > optimizations make sense.
> >
> > +1 for init containers or a more generalized way of obtaining
> >
> > arbitrary
> >
> > files. File fetching isn't specific to just SQL--it also matters for
> >
> > Java
> >
> > applications if the user doesn't want to rebuild a Flink image and
> >
> > just
> >
> > wants to modify the user application fat jar.
> >
> > Please note that we could reuse the checkpoint storage like S3/HDFS,
> >
> > which
> >
> > should
> >
> >
> > be required to run Flink in production, so I guess that would be
> >
> > acceptable
> >
> > for most
> >
> >
> > users. WDYT?
> >
> >
> > If you do go this route, it would be nice to support writing these
> >
> > files
> >
> > to
> >
> > S3/HDFS via Flink. This makes access control and policy management
> >
> > simpler.
> >
> >
> > Also, what do you think about prefixing the config options with
> > `sql-driver` instead of just `sql` to be more specific?
> >
> > Best,
> > Mason
> >
> > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
> >
> > <mailto:
> >
> > paullin3280@gmail.com>> wrote:
> >
> >
> > Hi Jark,
> >
> > Thanks for your input! Please see my comments inline.
> >
> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > DataStream API also doesn't provide a default main class for users,
> > why do we need to provide such one for SQL?
> >
> >
> > Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> >
> > submitted
> >
> > via Flink CLI which actually could be DataStream/Table jobs.
> >
> > I think a default main class would be user-friendly which eliminates
> >
> > the
> >
> > need
> > for users to write a main class as SQLRunner in Flink K8s operator
> >
> > [1].
> >
> >
> > I thought the proposed SqlDriver was a dedicated main class
> >
> > accepting
> >
> > SQL files, is
> >
> > that correct?
> >
> >
> > Both JSON plans and SQL files are accepted. SQL Gateway should use
> >
> > JSON
> >
> > plans,
> > while CLI users may use either JSON plans or SQL files.
> >
> > Please see the updated FLIP[2] for more details.
> >
> > Personally, I prefer the way of init containers which doesn't
> >
> > depend
> >
> > on
> >
> > additional components.
> > This can reduce the moving parts of a production environment.
> > Depending on a distributed file system makes the testing, demo, and
> >
> > local
> >
> > setup harder than init containers.
> >
> >
> > Please note that we could reuse the checkpoint storage like S3/HDFS,
> >
> > which
> >
> > should
> > be required to run Flink in production, so I guess that would be
> > acceptable for most
> > users. WDYT?
> >
> > WRT testing, demo, and local setups, I think we could support the
> >
> > local
> >
> > filesystem
> > scheme i.e. file://** as the state backends do. It works as long as
> >
> > SQL
> >
> > Gateway
> > and JobManager(or SQL Driver) can access the resource directory
> >
> > (specified
> >
> > via
> > `sql-gateway.application.storage-dir`).
> >
> > Thanks!
> >
> > [1]
> >
> >
> >
> >
> >
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> >
> > [2]
> >
> >
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >
> > [3]
> >
> >
> >
> >
> >
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
> >
> > Hi Paul,
> >
> > Thanks for your reply. I left my comments inline.
> >
> > As the FLIP said, it’s good to have a default main class for Flink
> >
> > SQLs,
> >
> > which allows users to submit Flink SQLs in the same way as
> >
> > DataStream
> >
> > jobs, or else users need to write their own main class.
> >
> >
> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > DataStream API also doesn't provide a default main class for users,
> > why do we need to provide such one for SQL?
> >
> > With the help of ExecNodeGraph, do we still need the serialized
> > SessionState? If not, we could make SQL Driver accepts two
> >
> > serialized
> >
> > formats:
> >
> >
> > No, ExecNodeGraph doesn't need to serialize SessionState. I thought
> >
> > the
> >
> > proposed SqlDriver was a dedicated main class accepting SQL files,
> >
> > is
> >
> > that correct?
> > If true, we have to ship the SessionState for this case which is a
> >
> > large
> >
> > work.
> > I think we just need a JsonPlanDriver which is a main class that
> >
> > accepts
> >
> > JsonPlan as the parameter.
> >
> >
> > The common solutions I know is to use distributed file systems or
> >
> > use
> >
> > init containers to localize the resources.
> >
> >
> > Personally, I prefer the way of init containers which doesn't
> >
> > depend
> >
> > on
> >
> > additional components.
> > This can reduce the moving parts of a production environment.
> > Depending on a distributed file system makes the testing, demo, and
> >
> > local
> >
> > setup harder than init containers.
> >
> > Best,
> > Jark
> >
> >
> >
> >
> > On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
> >
> > <mailto:
> >
> > paullin3280@gmail.com> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com>>> wrote:
> >
> >
> > The FLIP is in the early phase and some details are not included,
> >
> > but
> >
> > fortunately, we got lots of valuable ideas from the discussion.
> >
> > Thanks to everyone who joined the dissuasion!
> > @Weihua @Shanmon @Shengkai @Biao @Jark
> >
> > This weekend I’m gonna revisit and update the FLIP, adding more
> > details. Hopefully, we can further align our opinions.
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
> >
> > paullin3280@gmail.com>> 写道：
> >
> >
> > Hi Jark,
> >
> > Thanks a lot for your input!
> >
> > If we decide to submit ExecNodeGraph instead of SQL file, is it
> >
> > still
> >
> > necessary to support SQL Driver?
> >
> >
> > I think so. Apart from usage in SQL Gateway, SQL Driver could
> >
> > simplify
> >
> > Flink SQL execution with Flink CLI.
> >
> > As the FLIP said, it’s good to have a default main class for
> >
> > Flink
> >
> > SQLs,
> >
> > which allows users to submit Flink SQLs in the same way as
> >
> > DataStream
> >
> > jobs, or else users need to write their own main class.
> >
> > SQL Driver needs to serialize SessionState which is very
> >
> > challenging
> >
> > but not detailed covered in the FLIP.
> >
> >
> > With the help of ExecNodeGraph, do we still need the serialized
> > SessionState? If not, we could make SQL Driver accepts two
> >
> > serialized
> >
> > formats:
> >
> > - SQL files for user-facing public usage
> > - ExecNodeGraph for internal usage
> >
> > It’s kind of similar to the relationship between job jars and
> >
> > jobgraphs.
> >
> >
> > Regarding "K8S doesn't support shipping multiple jars", is that
> >
> > true?
> >
> > Is it
> >
> > possible to support it?
> >
> >
> > Yes, K8s doesn’t distribute any files. It’s the users’
> >
> > responsibility
> >
> > to
> >
> > make
> >
> > sure the resources are accessible in the containers. The common
> >
> > solutions
> >
> > I know is to use distributed file systems or use init containers
> >
> > to
> >
> > localize the
> >
> > resources.
> >
> > Now I lean toward introducing a fs to do the distribution job.
> >
> > WDYT?
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
> >
> > imjark@gmail.com>
> >
> > <mailto:imjark@gmail.com <ma...@gmail.com>>
> >
> > <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
> >
> > imjark@gmail.com <ma...@gmail.com>>>>
> >
> > 写道：
> >
> >
> > Hi Paul,
> >
> > Thanks for starting this discussion. I like the proposal! This
> >
> > is
> >
> > a
> >
> > frequently requested feature!
> >
> > I agree with Shengkai that ExecNodeGraph as the submission
> >
> > object
> >
> > is a
> >
> > better idea than SQL file. To be more specific, it should be
> >
> > JsonPlanGraph
> >
> > or CompiledPlan which is the serializable representation.
> >
> > CompiledPlan
> >
> > is a
> >
> > clear separation between compiling/optimization/validation and
> >
> > execution.
> >
> > This can keep the validation and metadata accessing still on the
> >
> > SQLGateway
> >
> > side. This allows SQLGateway to leverage some metadata caching
> >
> > and
> >
> > UDF
> >
> > JAR
> >
> > caching for better compiling performance.
> >
> > If we decide to submit ExecNodeGraph instead of SQL file, is it
> >
> > still
> >
> > necessary to support SQL Driver? Regarding non-interactive SQL
> >
> > jobs,
> >
> > users
> >
> > can use the Table API program for application mode. SQL Driver
> >
> > needs
> >
> > to
> >
> > serialize SessionState which is very challenging but not
> >
> > detailed
> >
> > covered
> >
> > in the FLIP.
> >
> > Regarding "K8S doesn't support shipping multiple jars", is that
> >
> > true?
> >
> > Is it
> >
> > possible to support it?
> >
> > Best,
> > Jark
> >
> >
> >
> > On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
> >
> > <ma...@gmail.com> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com> <mailto:
> >
> > paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
> >
> >
> > Hi Weihua,
> >
> > You’re right. Distributing the SQLs to the TMs is one of the
> >
> > challenging
> >
> > parts of this FLIP.
> >
> > Web submission is not enabled in application mode currently as
> >
> > you
> >
> > said,
> >
> > but it could be changed if we have good reasons.
> >
> > What do you think about introducing a distributed storage for
> >
> > SQL
> >
> > Gateway?
> >
> >
> > We could make use of Flink file systems [1] to distribute the
> >
> > SQL
> >
> > Gateway
> >
> > generated resources, that should solve the problem at its root
> >
> > cause.
> >
> >
> > Users could specify Flink-supported file systems to ship files.
> >
> > It’s
> >
> > only
> >
> > required when using SQL Gateway with K8s application mode.
> >
> > [1]
> >
> >
> >
> >
> >
> >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> > <
> >
> >
> >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >
> > <
> >
> >
> >
> >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> > <
> >
> >
> >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >
> >
> > <
> >
> >
> >
> >
> >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> > <
> >
> >
> >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >
> > <
> >
> >
> >
> >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> > <
> >
> >
> >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >
> >
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> >
> > huweihua.ckl@gmail.com> <mailto:
> >
> > huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
> >
> >
> > Thanks Paul for your reply.
> >
> > SQLDriver looks good to me.
> >
> > 2. Do you mean a pass the SQL string a configuration or a
> >
> > program
> >
> > argument?
> >
> >
> >
> > I brought this up because we were unable to pass the SQL file
> >
> > to
> >
> > Flink
> >
> > using Kubernetes mode.
> > For DataStream/Python users, they need to prepare their images
> >
> > for
> >
> > the
> >
> > jars
> >
> > and dependencies.
> > But for SQL users, they can use a common image to run
> >
> > different
> >
> > SQL
> >
> > queries
> >
> > if there are no other udf requirements.
> > It would be great if the SQL query and image were not bound.
> >
> > Using strings is a way to decouple these, but just as you
> >
> > mentioned,
> >
> > it's
> >
> > not easy to pass complex SQL.
> >
> > use web submission
> >
> > AFAIK, we can not use web submission in the Application mode.
> >
> > Please
> >
> > correct me if I'm wrong.
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Wed, May 31, 2023 at 9:37 PM Paul Lam <
> >
> > paullin3280@gmail.com
> >
> > <ma...@gmail.com>
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> >
> > wrote:
> >
> >
> > Hi Biao,
> >
> > Thanks for your comments!
> >
> > 1. Scope: is this FLIP only targeted for non-interactive
> >
> > Flink
> >
> > SQL
> >
> > jobs
> >
> > in
> >
> > Application mode? More specifically, if we use SQL
> >
> > client/gateway
> >
> > to
> >
> > execute some interactive SQLs like a SELECT query, can we
> >
> > ask
> >
> > flink
> >
> > to
> >
> > use
> >
> > Application mode to execute those queries after this FLIP?
> >
> >
> > Thanks for pointing it out. I think only DMLs would be
> >
> > executed
> >
> > via
> >
> > SQL
> >
> > Driver.
> > I'll add the scope to the FLIP.
> >
> > 2. Deployment: I believe in YARN mode, the implementation is
> >
> > trivial as
> >
> > we
> >
> > can ship files via YARN's tool easily but for K8s, things
> >
> > can
> >
> > be
> >
> > more
> >
> > complicated as Shengkai said.
> >
> >
> >
> > Your input is very informative. I’m thinking about using web
> >
> > submission,
> >
> > but it requires exposing the JobManager port which could also
> >
> > be
> >
> > a
> >
> > problem
> >
> > on K8s.
> >
> > Another approach is to explicitly require a distributed
> >
> > storage
> >
> > to
> >
> > ship
> >
> > files,
> > but we may need a new deployment executor for that.
> >
> > What do you think of these two approaches?
> >
> > 3. Serialization of SessionState: in SessionState, there are
> >
> > some
> >
> > unserializable fields
> > like
> >
> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >
> > It
> >
> > may be worthwhile to add more details about the
> >
> > serialization
> >
> > part.
> >
> >
> > I agree. That’s a missing part. But if we use ExecNodeGraph
> >
> > as
> >
> > Shengkai
> >
> > mentioned, do we eliminate the need for serialization of
> >
> > SessionState?
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
> >
> > biaogeng7@gmail.com> <mailto:
> >
> > biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
> >
> >
> > Thanks Paul for the proposal!I believe it would be very
> >
> > useful
> >
> > for
> >
> > flink
> >
> > users.
> > After reading the FLIP, I have some questions:
> > 1. Scope: is this FLIP only targeted for non-interactive
> >
> > Flink
> >
> > SQL
> >
> > jobs
> >
> > in
> >
> > Application mode? More specifically, if we use SQL
> >
> > client/gateway
> >
> > to
> >
> > execute some interactive SQLs like a SELECT query, can we
> >
> > ask
> >
> > flink
> >
> > to
> >
> > use
> >
> > Application mode to execute those queries after this FLIP?
> > 2. Deployment: I believe in YARN mode, the implementation is
> >
> > trivial as
> >
> > we
> >
> > can ship files via YARN's tool easily but for K8s, things
> >
> > can
> >
> > be
> >
> > more
> >
> > complicated as Shengkai said. I have implemented a simple
> >
> > POC
> >
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> > <
> >
> >
> >
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >
> > <
> >
> >
> >
> >
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> > <
> >
> >
> >
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >
> >
> >
> > based on SQL client before(i.e. consider the SQL client
> >
> > which
> >
> > supports
> >
> > executing a SQL file as the SQL driver in this FLIP). One
> >
> > problem
> >
> > I
> >
> > have
> >
> > met is how do we ship SQL files ( or Job Graph) to the k8s
> >
> > side.
> >
> > Without
> >
> > such support, users have to modify the initContainer or
> >
> > rebuild
> >
> > a
> >
> > new
> >
> > K8s
> >
> > image every time to fetch the SQL file. Like the flink k8s
> >
> > operator,
> >
> > one
> >
> > workaround is to utilize the flink config(transforming the
> >
> > SQL
> >
> > file
> >
> > to
> >
> > a
> >
> > escaped string like Weihua mentioned) which will be
> >
> > converted
> >
> > to a
> >
> > ConfigMap but K8s has size limit of ConfigMaps(no larger
> >
> > than
> >
> > 1MB
> >
> > <
> >
> > https://kubernetes.io/docs/concepts/configuration/configmap/
> >
> > <
> >
> > https://kubernetes.io/docs/concepts/configuration/configmap/> <
> >
> > https://kubernetes.io/docs/concepts/configuration/configmap/ <
> >
> > https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
> >
> > Not
> >
> > sure
> >
> > if we have better solutions.
> > 3. Serialization of SessionState: in SessionState, there are
> >
> > some
> >
> > unserializable fields
> > like
> >
> > org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >
> > It
> >
> > may be worthwhile to add more details about the
> >
> > serialization
> >
> > part.
> >
> >
> > Best,
> > Biao Geng
> >
> > Paul Lam <paullin3280@gmail.com <mailto:
> >
> > paullin3280@gmail.com
> >
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> >
> > 于2023年5月31日周三 11:49写道：
> >
> >
> > Hi Weihua,
> >
> > Thanks a lot for your input! Please see my comments inline.
> >
> > - Is SQLRunner the better name? We use this to run a SQL
> >
> > Job.
> >
> > (Not
> >
> > strong,
> >
> > the SQLDriver is fine for me)
> >
> >
> > I’ve thought about SQL Runner but picked SQL Driver for the
> >
> > following
> >
> > reasons FYI:
> >
> > 1. I have a PythonDriver doing the same job for PyFlink [1]
> > 2. Flink program's main class is sort of like Driver in
> >
> > JDBC
> >
> > which
> >
> > translates SQLs into
> > databases specific languages.
> >
> > In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >
> > - Could we run SQL jobs using SQL in strings? Otherwise,
> >
> > we
> >
> > need
> >
> > to
> >
> > prepare
> >
> > a SQL file in an image for Kubernetes application mode,
> >
> > which
> >
> > may
> >
> > be
> >
> > a
> >
> > bit
> >
> > cumbersome.
> >
> >
> > Do you mean a pass the SQL string a configuration or a
> >
> > program
> >
> > argument?
> >
> >
> > I thought it might be convenient for testing propose, but
> >
> > not
> >
> > recommended
> >
> > for production,
> > cause Flink SQLs could be complicated and involves lots of
> >
> > characters
> >
> > that
> >
> > need to escape.
> >
> > WDYT?
> >
> > - I noticed that we don't specify the SQLDriver jar in the
> >
> > "run-application"
> >
> > command. Does that mean we need to perform automatic
> >
> > detection
> >
> > in
> >
> > Flink?
> >
> >
> > Yes! It’s like running a PyFlink job with the following
> >
> > command:
> >
> >
> > ```
> > ./bin/flink run \
> > --pyModule table.word_count \
> > --pyFiles examples/python/table
> > ```
> >
> > The CLI determines if it’s a SQL job, if yes apply the SQL
> >
> > Driver
> >
> > automatically.
> >
> >
> > [1]
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> > <
> >
> >
> >
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >
> > <
> >
> >
> >
> >
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> > <
> >
> >
> >
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >
> >
> >
> > Best,
> > Paul Lam
> >
> > 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com
> >
> > <mailto:
> >
> > huweihua.ckl@gmail.com> <mailto:
> >
> > huweihua.ckl@gmail.com>> 写道：
> >
> >
> > Thanks Paul for the proposal.
> >
> > +1 for this. It is valuable in improving ease of use.
> >
> > I have a few questions.
> > - Is SQLRunner the better name? We use this to run a SQL
> >
> > Job.
> >
> > (Not
> >
> > strong,
> >
> > the SQLDriver is fine for me)
> > - Could we run SQL jobs using SQL in strings? Otherwise,
> >
> > we
> >
> > need
> >
> > to
> >
> > prepare
> >
> > a SQL file in an image for Kubernetes application mode,
> >
> > which
> >
> > may
> >
> > be
> >
> > a
> >
> > bit
> >
> > cumbersome.
> > - I noticed that we don't specify the SQLDriver jar in the
> >
> > "run-application"
> >
> > command. Does that mean we need to perform automatic
> >
> > detection
> >
> > in
> >
> > Flink?
> >
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> >
> > paullin3280@gmail.com
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> >
> > wrote:
> >
> >
> > Hi team,
> >
> > I’d like to start a discussion about FLIP-316 [1], which
> >
> > introduces
> >
> > a
> >
> > SQL
> >
> > driver as the
> > default main class for Flink SQL jobs.
> >
> > Currently, Flink SQL could be executed out of the box
> >
> > either
> >
> > via
> >
> > SQL
> >
> > Client/Gateway
> > or embedded in a Flink Java/Python program.
> >
> > However, each one has its drawback:
> >
> > - SQL Client/Gateway doesn’t support the application
> >
> > deployment
> >
> > mode
> >
> > [2]
> >
> > - Flink Java/Python program requires extra work to write
> >
> > a
> >
> > non-SQL
> >
> > program
> >
> >
> > Therefore, I propose adding a SQL driver to act as the
> >
> > default
> >
> > main
> >
> > class
> >
> > for SQL jobs.
> > Please see the FLIP docs for details and feel free to
> >
> > comment.
> >
> > Thanks!
> >
> >
> > [1]
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >
> >
> > [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> > https://issues.apache.org/jira/browse/FLINK-26541>
> >
> > Best,
> > Paul Lam
> >
> >
> >
> >
> >
> >
> >
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Yang,
Thanks a lot for your input!
It’s great that FLINK-28915 has covered the file download part. I’ve created
a ticket for the file upload part [1]. It's a prerequisite for supporting K8s application
mode for SQL Gateway.
[1] https://issues.apache.org/jira/browse/FLINK-32315
Best,
Paul Lam
> 2023年6月12日 11:40，Yang Wang <wa...@apache.org> 写道：
>
> Sorry for the late reply. I am in favor of introducing such a built-in
> resource localization mechanism
> based on Flink FileSystem. Then FLINK-28915[1] could be the second step
> which will download
> the jars and dependencies to the JobManager/TaskManager local directory
> before working.
>
> The first step could be done in another ticket in Flink. Or some external
> Flink jobs management system
> could also take care of this.
>
> [1]. https://issues.apache.org/jira/browse/FLINK-28915
>
> Best,
> Yang
>
> Paul Lam <pa...@gmail.com> 于2023年6月9日周五 17:39写道：
>
>> Hi Mason,
>>
>> I get your point. I'm increasingly feeling the need to introduce a
>> built-in
>> file distribution mechanism for flink-kubernetes module, just like Spark
>> does with `spark.kubernetes.file.upload.path` [1].
>>
>> I’m assuming the workflow is as follows:
>>
>> - KubernetesClusterDescripter uploads all local resources to a remote
>> storage via Flink filesystem (skips if the resources are already remote).
>> - KubernetesApplicationClusterEntrypoint downloads the resources
>> and put them in the classpath during startup.
>>
>> I wouldn't mind splitting it into another FLIP to ensure that everything is
>> done correctly.
>>
>> cc'ed @Yang to gather more opinions.
>>
>> [1]
>> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
>>
>> Best,
>> Paul Lam
>>
>> 2023年6月8日 12:15，Mason Chen <ma...@gmail.com> 写道：
>>
>> Hi Paul,
>>
>> Thanks for your response!
>>
>> I agree that utilizing SQL Drivers in Java applications is equally
>> important
>>
>> as employing them in SQL Gateway. WRT init containers, I think most
>> users use them just as a workaround. For example, wget a jar from the
>> maven repo.
>>
>> We could implement the functionality in SQL Driver in a more graceful
>> way and the flink-supported filesystem approach seems to be a
>> good choice.
>>
>>
>> My main point is: can we solve the problem with a design agnostic of SQL
>> and Stream API? I mentioned a use case where this ability is useful for
>> Java or Stream API applications. Maybe this is even a non-goal to your FLIP
>> since you are focusing on the driver entrypoint.
>>
>> Jark mentioned some optimizations:
>>
>> This allows SQLGateway to leverage some metadata caching and UDF JAR
>> caching for better compiling performance.
>>
>> It would be great to see this even outside the SQLGateway (i.e. UDF JAR
>> caching).
>>
>> Best,
>> Mason
>>
>> On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fs...@gmail.com> wrote:
>>
>> Hi. Paul. Thanks for your update and the update makes me understand the
>> design much better.
>>
>> But I still have some questions about the FLIP.
>>
>> For SQL Gateway, only DMLs need to be delegated to the SQL server
>> Driver. I would think about the details and update the FLIP. Do you have
>>
>> some
>>
>> ideas already?
>>
>>
>> If the applicaiton mode can not support library mode, I think we should
>> only execute INSERT INTO and UPDATE/ DELETE statement in the application
>> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
>> statements. The ANALYZE TABLE syntax need to register the statistic to the
>> catalog after job finishes and the CALL PROCEDURE statement doesn't
>> generate the ExecNodeGraph.
>>
>> * Introduce storage via option `sql-gateway.application.storage-dir`
>>
>> If we can not support to submit the jars through web submission, +1 to
>> introduce the options to upload the files. While I think the uploader
>> should be responsible to remove the uploaded jars. Can we remove the jars
>> if the job is running or gateway exits?
>>
>> * JobID is not avaliable
>>
>> Can we use the returned rest client by ApplicationDeployer to query the job
>> id? I am concerned that users don't know which job is related to the
>> submitted SQL.
>>
>> * Do we need to introduce a new module named flink-table-sql-runner?
>>
>> It seems we need to introduce a new module. Will the new module is
>> available in the distribution package? I agree with Jark that we don't need
>> to introduce this for table-API users and these users have their main
>> class. If we want to make users write the k8s operator more easily, I think
>> we should modify the k8s operator repo. If we don't need to support SQL
>> files, can we make this jar only visible in the sql-gateway like we do in
>> the planner loader?[1]
>>
>> [1]
>>
>>
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
>>
>> Best,
>> Shengkai
>>
>>
>>
>>
>>
>>
>>
>>
>> Weihua Hu <hu...@gmail.com> 于2023年6月7日周三 10:52写道：
>>
>> Hi,
>>
>> Thanks for updating the FLIP.
>>
>> I have two cents on the distribution of SQLs and resources.
>> 1. Should we support a common file distribution mechanism for k8s
>> application mode?
>> I have seen some issues and requirements on the mailing list.
>> In our production environment, we implement the download command in the
>> CliFrontend.
>> And automatically add an init container to the POD for file
>>
>> downloading.
>>
>> The advantage of this
>> is that we can use all Flink-supported file systems to store files.
>>
>> This need more discussion. I would appreciate hearing more opinions.
>>
>> 2. In this FLIP, we distribute files in two different ways in YARN and
>> Kubernetes. Can we combine it in one way?
>> If we don't want to implement a common file distribution for k8s
>> application mode. Could we use the SQLDriver
>> to download the files both in YARN and K8S? IMO, this can reduce the
>>
>> cost
>>
>> of code maintenance.
>>
>> Best,
>> Weihua
>>
>>
>> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
>>
>> Hi Mason,
>>
>> Thanks for your input!
>>
>> +1 for init containers or a more generalized way of obtaining
>>
>> arbitrary
>>
>> files. File fetching isn't specific to just SQL--it also matters for
>>
>> Java
>>
>> applications if the user doesn't want to rebuild a Flink image and
>>
>> just
>>
>> wants to modify the user application fat jar.
>>
>>
>> I agree that utilizing SQL Drivers in Java applications is equally
>> important
>> as employing them in SQL Gateway. WRT init containers, I think most
>> users use them just as a workaround. For example, wget a jar from the
>> maven repo.
>>
>> We could implement the functionality in SQL Driver in a more graceful
>> way and the flink-supported filesystem approach seems to be a
>> good choice.
>>
>> Also, what do you think about prefixing the config options with
>> `sql-driver` instead of just `sql` to be more specific?
>>
>>
>> LGTM, since SQL Driver is a public interface and the options are
>> specific to it.
>>
>> Best,
>> Paul Lam
>>
>> 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
>>
>> Hi Paul,
>>
>> +1 for this feature and supporting SQL file + JSON plans. We get a
>>
>> lot
>>
>> of
>>
>> requests to just be able to submit a SQL file, but the JSON plan
>> optimizations make sense.
>>
>> +1 for init containers or a more generalized way of obtaining
>>
>> arbitrary
>>
>> files. File fetching isn't specific to just SQL--it also matters for
>>
>> Java
>>
>> applications if the user doesn't want to rebuild a Flink image and
>>
>> just
>>
>> wants to modify the user application fat jar.
>>
>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>
>> which
>>
>> should
>>
>>
>> be required to run Flink in production, so I guess that would be
>>
>> acceptable
>>
>> for most
>>
>>
>> users. WDYT?
>>
>>
>> If you do go this route, it would be nice to support writing these
>>
>> files
>>
>> to
>>
>> S3/HDFS via Flink. This makes access control and policy management
>>
>> simpler.
>>
>>
>> Also, what do you think about prefixing the config options with
>> `sql-driver` instead of just `sql` to be more specific?
>>
>> Best,
>> Mason
>>
>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
>>
>> <mailto:
>>
>> paullin3280@gmail.com>> wrote:
>>
>>
>> Hi Jark,
>>
>> Thanks for your input! Please see my comments inline.
>>
>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>> DataStream API also doesn't provide a default main class for users,
>> why do we need to provide such one for SQL?
>>
>>
>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>>
>> submitted
>>
>> via Flink CLI which actually could be DataStream/Table jobs.
>>
>> I think a default main class would be user-friendly which eliminates
>>
>> the
>>
>> need
>> for users to write a main class as SQLRunner in Flink K8s operator
>>
>> [1].
>>
>>
>> I thought the proposed SqlDriver was a dedicated main class
>>
>> accepting
>>
>> SQL files, is
>>
>> that correct?
>>
>>
>> Both JSON plans and SQL files are accepted. SQL Gateway should use
>>
>> JSON
>>
>> plans,
>> while CLI users may use either JSON plans or SQL files.
>>
>> Please see the updated FLIP[2] for more details.
>>
>> Personally, I prefer the way of init containers which doesn't
>>
>> depend
>>
>> on
>>
>> additional components.
>> This can reduce the moving parts of a production environment.
>> Depending on a distributed file system makes the testing, demo, and
>>
>> local
>>
>> setup harder than init containers.
>>
>>
>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>
>> which
>>
>> should
>> be required to run Flink in production, so I guess that would be
>> acceptable for most
>> users. WDYT?
>>
>> WRT testing, demo, and local setups, I think we could support the
>>
>> local
>>
>> filesystem
>> scheme i.e. file://** as the state backends do. It works as long as
>>
>> SQL
>>
>> Gateway
>> and JobManager(or SQL Driver) can access the resource directory
>>
>> (specified
>>
>> via
>> `sql-gateway.application.storage-dir`).
>>
>> Thanks!
>>
>> [1]
>>
>>
>>
>>
>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
>>
>> [2]
>>
>>
>>
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>
>> [3]
>>
>>
>>
>>
>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>>
>>
>> Best,
>> Paul Lam
>>
>> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
>>
>> Hi Paul,
>>
>> Thanks for your reply. I left my comments inline.
>>
>> As the FLIP said, it’s good to have a default main class for Flink
>>
>> SQLs,
>>
>> which allows users to submit Flink SQLs in the same way as
>>
>> DataStream
>>
>> jobs, or else users need to write their own main class.
>>
>>
>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>> DataStream API also doesn't provide a default main class for users,
>> why do we need to provide such one for SQL?
>>
>> With the help of ExecNodeGraph, do we still need the serialized
>> SessionState? If not, we could make SQL Driver accepts two
>>
>> serialized
>>
>> formats:
>>
>>
>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
>>
>> the
>>
>> proposed SqlDriver was a dedicated main class accepting SQL files,
>>
>> is
>>
>> that correct?
>> If true, we have to ship the SessionState for this case which is a
>>
>> large
>>
>> work.
>> I think we just need a JsonPlanDriver which is a main class that
>>
>> accepts
>>
>> JsonPlan as the parameter.
>>
>>
>> The common solutions I know is to use distributed file systems or
>>
>> use
>>
>> init containers to localize the resources.
>>
>>
>> Personally, I prefer the way of init containers which doesn't
>>
>> depend
>>
>> on
>>
>> additional components.
>> This can reduce the moving parts of a production environment.
>> Depending on a distributed file system makes the testing, demo, and
>>
>> local
>>
>> setup harder than init containers.
>>
>> Best,
>> Jark
>>
>>
>>
>>
>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
>>
>> <mailto:
>>
>> paullin3280@gmail.com> <mailto:
>>
>> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>>
>>
>> The FLIP is in the early phase and some details are not included,
>>
>> but
>>
>> fortunately, we got lots of valuable ideas from the discussion.
>>
>> Thanks to everyone who joined the dissuasion!
>> @Weihua @Shanmon @Shengkai @Biao @Jark
>>
>> This weekend I’m gonna revisit and update the FLIP, adding more
>> details. Hopefully, we can further align our opinions.
>>
>> Best,
>> Paul Lam
>>
>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
>>
>> paullin3280@gmail.com>> 写道：
>>
>>
>> Hi Jark,
>>
>> Thanks a lot for your input!
>>
>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>
>> still
>>
>> necessary to support SQL Driver?
>>
>>
>> I think so. Apart from usage in SQL Gateway, SQL Driver could
>>
>> simplify
>>
>> Flink SQL execution with Flink CLI.
>>
>> As the FLIP said, it’s good to have a default main class for
>>
>> Flink
>>
>> SQLs,
>>
>> which allows users to submit Flink SQLs in the same way as
>>
>> DataStream
>>
>> jobs, or else users need to write their own main class.
>>
>> SQL Driver needs to serialize SessionState which is very
>>
>> challenging
>>
>> but not detailed covered in the FLIP.
>>
>>
>> With the help of ExecNodeGraph, do we still need the serialized
>> SessionState? If not, we could make SQL Driver accepts two
>>
>> serialized
>>
>> formats:
>>
>> - SQL files for user-facing public usage
>> - ExecNodeGraph for internal usage
>>
>> It’s kind of similar to the relationship between job jars and
>>
>> jobgraphs.
>>
>>
>> Regarding "K8S doesn't support shipping multiple jars", is that
>>
>> true?
>>
>> Is it
>>
>> possible to support it?
>>
>>
>> Yes, K8s doesn’t distribute any files. It’s the users’
>>
>> responsibility
>>
>> to
>>
>> make
>>
>> sure the resources are accessible in the containers. The common
>>
>> solutions
>>
>> I know is to use distributed file systems or use init containers
>>
>> to
>>
>> localize the
>>
>> resources.
>>
>> Now I lean toward introducing a fs to do the distribution job.
>>
>> WDYT?
>>
>>
>> Best,
>> Paul Lam
>>
>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
>>
>> imjark@gmail.com>
>>
>> <mailto:imjark@gmail.com <ma...@gmail.com>>
>>
>> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
>>
>> imjark@gmail.com <ma...@gmail.com>>>>
>>
>> 写道：
>>
>>
>> Hi Paul,
>>
>> Thanks for starting this discussion. I like the proposal! This
>>
>> is
>>
>> a
>>
>> frequently requested feature!
>>
>> I agree with Shengkai that ExecNodeGraph as the submission
>>
>> object
>>
>> is a
>>
>> better idea than SQL file. To be more specific, it should be
>>
>> JsonPlanGraph
>>
>> or CompiledPlan which is the serializable representation.
>>
>> CompiledPlan
>>
>> is a
>>
>> clear separation between compiling/optimization/validation and
>>
>> execution.
>>
>> This can keep the validation and metadata accessing still on the
>>
>> SQLGateway
>>
>> side. This allows SQLGateway to leverage some metadata caching
>>
>> and
>>
>> UDF
>>
>> JAR
>>
>> caching for better compiling performance.
>>
>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>
>> still
>>
>> necessary to support SQL Driver? Regarding non-interactive SQL
>>
>> jobs,
>>
>> users
>>
>> can use the Table API program for application mode. SQL Driver
>>
>> needs
>>
>> to
>>
>> serialize SessionState which is very challenging but not
>>
>> detailed
>>
>> covered
>>
>> in the FLIP.
>>
>> Regarding "K8S doesn't support shipping multiple jars", is that
>>
>> true?
>>
>> Is it
>>
>> possible to support it?
>>
>> Best,
>> Jark
>>
>>
>>
>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
>>
>> <ma...@gmail.com> <mailto:
>>
>> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>>
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:
>>
>> paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>>
>>
>> Hi Weihua,
>>
>> You’re right. Distributing the SQLs to the TMs is one of the
>>
>> challenging
>>
>> parts of this FLIP.
>>
>> Web submission is not enabled in application mode currently as
>>
>> you
>>
>> said,
>>
>> but it could be changed if we have good reasons.
>>
>> What do you think about introducing a distributed storage for
>>
>> SQL
>>
>> Gateway?
>>
>>
>> We could make use of Flink file systems [1] to distribute the
>>
>> SQL
>>
>> Gateway
>>
>> generated resources, that should solve the problem at its root
>>
>> cause.
>>
>>
>> Users could specify Flink-supported file systems to ship files.
>>
>> It’s
>>
>> only
>>
>> required when using SQL Gateway with K8s application mode.
>>
>> [1]
>>
>>
>>
>>
>>
>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>> <
>>
>>
>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>>
>> <
>>
>>
>>
>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>> <
>>
>>
>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>>
>>
>> <
>>
>>
>>
>>
>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>> <
>>
>>
>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>>
>> <
>>
>>
>>
>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>> <
>>
>>
>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>>
>>
>>
>>
>> Best,
>> Paul Lam
>>
>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
>>
>> huweihua.ckl@gmail.com> <mailto:
>>
>> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
>>
>>
>> Thanks Paul for your reply.
>>
>> SQLDriver looks good to me.
>>
>> 2. Do you mean a pass the SQL string a configuration or a
>>
>> program
>>
>> argument?
>>
>>
>>
>> I brought this up because we were unable to pass the SQL file
>>
>> to
>>
>> Flink
>>
>> using Kubernetes mode.
>> For DataStream/Python users, they need to prepare their images
>>
>> for
>>
>> the
>>
>> jars
>>
>> and dependencies.
>> But for SQL users, they can use a common image to run
>>
>> different
>>
>> SQL
>>
>> queries
>>
>> if there are no other udf requirements.
>> It would be great if the SQL query and image were not bound.
>>
>> Using strings is a way to decouple these, but just as you
>>
>> mentioned,
>>
>> it's
>>
>> not easy to pass complex SQL.
>>
>> use web submission
>>
>> AFAIK, we can not use web submission in the Application mode.
>>
>> Please
>>
>> correct me if I'm wrong.
>>
>>
>> Best,
>> Weihua
>>
>>
>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <
>>
>> paullin3280@gmail.com
>>
>> <ma...@gmail.com>
>>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>
>> wrote:
>>
>>
>> Hi Biao,
>>
>> Thanks for your comments!
>>
>> 1. Scope: is this FLIP only targeted for non-interactive
>>
>> Flink
>>
>> SQL
>>
>> jobs
>>
>> in
>>
>> Application mode? More specifically, if we use SQL
>>
>> client/gateway
>>
>> to
>>
>> execute some interactive SQLs like a SELECT query, can we
>>
>> ask
>>
>> flink
>>
>> to
>>
>> use
>>
>> Application mode to execute those queries after this FLIP?
>>
>>
>> Thanks for pointing it out. I think only DMLs would be
>>
>> executed
>>
>> via
>>
>> SQL
>>
>> Driver.
>> I'll add the scope to the FLIP.
>>
>> 2. Deployment: I believe in YARN mode, the implementation is
>>
>> trivial as
>>
>> we
>>
>> can ship files via YARN's tool easily but for K8s, things
>>
>> can
>>
>> be
>>
>> more
>>
>> complicated as Shengkai said.
>>
>>
>>
>> Your input is very informative. I’m thinking about using web
>>
>> submission,
>>
>> but it requires exposing the JobManager port which could also
>>
>> be
>>
>> a
>>
>> problem
>>
>> on K8s.
>>
>> Another approach is to explicitly require a distributed
>>
>> storage
>>
>> to
>>
>> ship
>>
>> files,
>> but we may need a new deployment executor for that.
>>
>> What do you think of these two approaches?
>>
>> 3. Serialization of SessionState: in SessionState, there are
>>
>> some
>>
>> unserializable fields
>> like
>>
>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>
>> It
>>
>> may be worthwhile to add more details about the
>>
>> serialization
>>
>> part.
>>
>>
>> I agree. That’s a missing part. But if we use ExecNodeGraph
>>
>> as
>>
>> Shengkai
>>
>> mentioned, do we eliminate the need for serialization of
>>
>> SessionState?
>>
>>
>> Best,
>> Paul Lam
>>
>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
>>
>> biaogeng7@gmail.com> <mailto:
>>
>> biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
>>
>>
>> Thanks Paul for the proposal!I believe it would be very
>>
>> useful
>>
>> for
>>
>> flink
>>
>> users.
>> After reading the FLIP, I have some questions:
>> 1. Scope: is this FLIP only targeted for non-interactive
>>
>> Flink
>>
>> SQL
>>
>> jobs
>>
>> in
>>
>> Application mode? More specifically, if we use SQL
>>
>> client/gateway
>>
>> to
>>
>> execute some interactive SQLs like a SELECT query, can we
>>
>> ask
>>
>> flink
>>
>> to
>>
>> use
>>
>> Application mode to execute those queries after this FLIP?
>> 2. Deployment: I believe in YARN mode, the implementation is
>>
>> trivial as
>>
>> we
>>
>> can ship files via YARN's tool easily but for K8s, things
>>
>> can
>>
>> be
>>
>> more
>>
>> complicated as Shengkai said. I have implemented a simple
>>
>> POC
>>
>> <
>>
>>
>>
>>
>>
>>
>>
>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>
>> <
>>
>>
>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>
>>
>> <
>>
>>
>>
>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>
>> <
>>
>>
>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>
>>
>>
>>
>> based on SQL client before(i.e. consider the SQL client
>>
>> which
>>
>> supports
>>
>> executing a SQL file as the SQL driver in this FLIP). One
>>
>> problem
>>
>> I
>>
>> have
>>
>> met is how do we ship SQL files ( or Job Graph) to the k8s
>>
>> side.
>>
>> Without
>>
>> such support, users have to modify the initContainer or
>>
>> rebuild
>>
>> a
>>
>> new
>>
>> K8s
>>
>> image every time to fetch the SQL file. Like the flink k8s
>>
>> operator,
>>
>> one
>>
>> workaround is to utilize the flink config(transforming the
>>
>> SQL
>>
>> file
>>
>> to
>>
>> a
>>
>> escaped string like Weihua mentioned) which will be
>>
>> converted
>>
>> to a
>>
>> ConfigMap but K8s has size limit of ConfigMaps(no larger
>>
>> than
>>
>> 1MB
>>
>> <
>>
>> https://kubernetes.io/docs/concepts/configuration/configmap/
>>
>> <
>>
>> https://kubernetes.io/docs/concepts/configuration/configmap/> <
>>
>> https://kubernetes.io/docs/concepts/configuration/configmap/ <
>>
>> https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
>>
>> Not
>>
>> sure
>>
>> if we have better solutions.
>> 3. Serialization of SessionState: in SessionState, there are
>>
>> some
>>
>> unserializable fields
>> like
>>
>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>
>> It
>>
>> may be worthwhile to add more details about the
>>
>> serialization
>>
>> part.
>>
>>
>> Best,
>> Biao Geng
>>
>> Paul Lam <paullin3280@gmail.com <mailto:
>>
>> paullin3280@gmail.com
>>
>>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>
>> 于2023年5月31日周三 11:49写道：
>>
>>
>> Hi Weihua,
>>
>> Thanks a lot for your input! Please see my comments inline.
>>
>> - Is SQLRunner the better name? We use this to run a SQL
>>
>> Job.
>>
>> (Not
>>
>> strong,
>>
>> the SQLDriver is fine for me)
>>
>>
>> I’ve thought about SQL Runner but picked SQL Driver for the
>>
>> following
>>
>> reasons FYI:
>>
>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>> 2. Flink program's main class is sort of like Driver in
>>
>> JDBC
>>
>> which
>>
>> translates SQLs into
>> databases specific languages.
>>
>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>
>> - Could we run SQL jobs using SQL in strings? Otherwise,
>>
>> we
>>
>> need
>>
>> to
>>
>> prepare
>>
>> a SQL file in an image for Kubernetes application mode,
>>
>> which
>>
>> may
>>
>> be
>>
>> a
>>
>> bit
>>
>> cumbersome.
>>
>>
>> Do you mean a pass the SQL string a configuration or a
>>
>> program
>>
>> argument?
>>
>>
>> I thought it might be convenient for testing propose, but
>>
>> not
>>
>> recommended
>>
>> for production,
>> cause Flink SQLs could be complicated and involves lots of
>>
>> characters
>>
>> that
>>
>> need to escape.
>>
>> WDYT?
>>
>> - I noticed that we don't specify the SQLDriver jar in the
>>
>> "run-application"
>>
>> command. Does that mean we need to perform automatic
>>
>> detection
>>
>> in
>>
>> Flink?
>>
>>
>> Yes! It’s like running a PyFlink job with the following
>>
>> command:
>>
>>
>> ```
>> ./bin/flink run \
>> --pyModule table.word_count \
>> --pyFiles examples/python/table
>> ```
>>
>> The CLI determines if it’s a SQL job, if yes apply the SQL
>>
>> Driver
>>
>> automatically.
>>
>>
>> [1]
>>
>>
>>
>>
>>
>>
>>
>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>
>> <
>>
>>
>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>
>>
>> <
>>
>>
>>
>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>
>> <
>>
>>
>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>
>>
>>
>>
>> Best,
>> Paul Lam
>>
>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com
>>
>> <mailto:
>>
>> huweihua.ckl@gmail.com> <mailto:
>>
>> huweihua.ckl@gmail.com>> 写道：
>>
>>
>> Thanks Paul for the proposal.
>>
>> +1 for this. It is valuable in improving ease of use.
>>
>> I have a few questions.
>> - Is SQLRunner the better name? We use this to run a SQL
>>
>> Job.
>>
>> (Not
>>
>> strong,
>>
>> the SQLDriver is fine for me)
>> - Could we run SQL jobs using SQL in strings? Otherwise,
>>
>> we
>>
>> need
>>
>> to
>>
>> prepare
>>
>> a SQL file in an image for Kubernetes application mode,
>>
>> which
>>
>> may
>>
>> be
>>
>> a
>>
>> bit
>>
>> cumbersome.
>> - I noticed that we don't specify the SQLDriver jar in the
>>
>> "run-application"
>>
>> command. Does that mean we need to perform automatic
>>
>> detection
>>
>> in
>>
>> Flink?
>>
>>
>>
>> Best,
>> Weihua
>>
>>
>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>>
>> paullin3280@gmail.com
>>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>
>> wrote:
>>
>>
>> Hi team,
>>
>> I’d like to start a discussion about FLIP-316 [1], which
>>
>> introduces
>>
>> a
>>
>> SQL
>>
>> driver as the
>> default main class for Flink SQL jobs.
>>
>> Currently, Flink SQL could be executed out of the box
>>
>> either
>>
>> via
>>
>> SQL
>>
>> Client/Gateway
>> or embedded in a Flink Java/Python program.
>>
>> However, each one has its drawback:
>>
>> - SQL Client/Gateway doesn’t support the application
>>
>> deployment
>>
>> mode
>>
>> [2]
>>
>> - Flink Java/Python program requires extra work to write
>>
>> a
>>
>> non-SQL
>>
>> program
>>
>>
>> Therefore, I propose adding a SQL driver to act as the
>>
>> default
>>
>> main
>>
>> class
>>
>> for SQL jobs.
>> Please see the FLIP docs for details and feel free to
>>
>> comment.
>>
>> Thanks!
>>
>>
>> [1]
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>
>> <
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>
>>
>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>> https://issues.apache.org/jira/browse/FLINK-26541>
>>
>> Best,
>> Paul Lam
>>
>>
>>
>>
>>
>>
>>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Yang Wang <wa...@apache.org>.
Sorry for the late reply. I am in favor of introducing such a built-in
resource localization mechanism
based on Flink FileSystem. Then FLINK-28915[1] could be the second step
which will download
the jars and dependencies to the JobManager/TaskManager local directory
before working.
The first step could be done in another ticket in Flink. Or some external
Flink jobs management system
could also take care of this.
[1]. https://issues.apache.org/jira/browse/FLINK-28915
Best,
Yang
Paul Lam <pa...@gmail.com> 于2023年6月9日周五 17:39写道：
> Hi Mason,
>
> I get your point. I'm increasingly feeling the need to introduce a
> built-in
> file distribution mechanism for flink-kubernetes module, just like Spark
> does with `spark.kubernetes.file.upload.path` [1].
>
> I’m assuming the workflow is as follows:
>
> - KubernetesClusterDescripter uploads all local resources to a remote
> storage via Flink filesystem (skips if the resources are already remote).
> - KubernetesApplicationClusterEntrypoint downloads the resources
> and put them in the classpath during startup.
>
> I wouldn't mind splitting it into another FLIP to ensure that everything is
> done correctly.
>
> cc'ed @Yang to gather more opinions.
>
> [1]
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
>
> Best,
> Paul Lam
>
> 2023年6月8日 12:15，Mason Chen <ma...@gmail.com> 写道：
>
> Hi Paul,
>
> Thanks for your response!
>
> I agree that utilizing SQL Drivers in Java applications is equally
> important
>
> as employing them in SQL Gateway. WRT init containers, I think most
> users use them just as a workaround. For example, wget a jar from the
> maven repo.
>
> We could implement the functionality in SQL Driver in a more graceful
> way and the flink-supported filesystem approach seems to be a
> good choice.
>
>
> My main point is: can we solve the problem with a design agnostic of SQL
> and Stream API? I mentioned a use case where this ability is useful for
> Java or Stream API applications. Maybe this is even a non-goal to your FLIP
> since you are focusing on the driver entrypoint.
>
> Jark mentioned some optimizations:
>
> This allows SQLGateway to leverage some metadata caching and UDF JAR
> caching for better compiling performance.
>
> It would be great to see this even outside the SQLGateway (i.e. UDF JAR
> caching).
>
> Best,
> Mason
>
> On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fs...@gmail.com> wrote:
>
> Hi. Paul. Thanks for your update and the update makes me understand the
> design much better.
>
> But I still have some questions about the FLIP.
>
> For SQL Gateway, only DMLs need to be delegated to the SQL server
> Driver. I would think about the details and update the FLIP. Do you have
>
> some
>
> ideas already?
>
>
> If the applicaiton mode can not support library mode, I think we should
> only execute INSERT INTO and UPDATE/ DELETE statement in the application
> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> statements. The ANALYZE TABLE syntax need to register the statistic to the
> catalog after job finishes and the CALL PROCEDURE statement doesn't
> generate the ExecNodeGraph.
>
> * Introduce storage via option `sql-gateway.application.storage-dir`
>
> If we can not support to submit the jars through web submission, +1 to
> introduce the options to upload the files. While I think the uploader
> should be responsible to remove the uploaded jars. Can we remove the jars
> if the job is running or gateway exits?
>
> * JobID is not avaliable
>
> Can we use the returned rest client by ApplicationDeployer to query the job
> id? I am concerned that users don't know which job is related to the
> submitted SQL.
>
> * Do we need to introduce a new module named flink-table-sql-runner?
>
> It seems we need to introduce a new module. Will the new module is
> available in the distribution package? I agree with Jark that we don't need
> to introduce this for table-API users and these users have their main
> class. If we want to make users write the k8s operator more easily, I think
> we should modify the k8s operator repo. If we don't need to support SQL
> files, can we make this jar only visible in the sql-gateway like we do in
> the planner loader?[1]
>
> [1]
>
>
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
>
> Best,
> Shengkai
>
>
>
>
>
>
>
>
> Weihua Hu <hu...@gmail.com> 于2023年6月7日周三 10:52写道：
>
> Hi,
>
> Thanks for updating the FLIP.
>
> I have two cents on the distribution of SQLs and resources.
> 1. Should we support a common file distribution mechanism for k8s
> application mode?
> I have seen some issues and requirements on the mailing list.
> In our production environment, we implement the download command in the
> CliFrontend.
> And automatically add an init container to the POD for file
>
> downloading.
>
> The advantage of this
> is that we can use all Flink-supported file systems to store files.
>
> This need more discussion. I would appreciate hearing more opinions.
>
> 2. In this FLIP, we distribute files in two different ways in YARN and
> Kubernetes. Can we combine it in one way?
> If we don't want to implement a common file distribution for k8s
> application mode. Could we use the SQLDriver
> to download the files both in YARN and K8S? IMO, this can reduce the
>
> cost
>
> of code maintenance.
>
> Best,
> Weihua
>
>
> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
>
> Hi Mason,
>
> Thanks for your input!
>
> +1 for init containers or a more generalized way of obtaining
>
> arbitrary
>
> files. File fetching isn't specific to just SQL--it also matters for
>
> Java
>
> applications if the user doesn't want to rebuild a Flink image and
>
> just
>
> wants to modify the user application fat jar.
>
>
> I agree that utilizing SQL Drivers in Java applications is equally
> important
> as employing them in SQL Gateway. WRT init containers, I think most
> users use them just as a workaround. For example, wget a jar from the
> maven repo.
>
> We could implement the functionality in SQL Driver in a more graceful
> way and the flink-supported filesystem approach seems to be a
> good choice.
>
> Also, what do you think about prefixing the config options with
> `sql-driver` instead of just `sql` to be more specific?
>
>
> LGTM, since SQL Driver is a public interface and the options are
> specific to it.
>
> Best,
> Paul Lam
>
> 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
>
> Hi Paul,
>
> +1 for this feature and supporting SQL file + JSON plans. We get a
>
> lot
>
> of
>
> requests to just be able to submit a SQL file, but the JSON plan
> optimizations make sense.
>
> +1 for init containers or a more generalized way of obtaining
>
> arbitrary
>
> files. File fetching isn't specific to just SQL--it also matters for
>
> Java
>
> applications if the user doesn't want to rebuild a Flink image and
>
> just
>
> wants to modify the user application fat jar.
>
> Please note that we could reuse the checkpoint storage like S3/HDFS,
>
> which
>
> should
>
>
> be required to run Flink in production, so I guess that would be
>
> acceptable
>
> for most
>
>
> users. WDYT?
>
>
> If you do go this route, it would be nice to support writing these
>
> files
>
> to
>
> S3/HDFS via Flink. This makes access control and policy management
>
> simpler.
>
>
> Also, what do you think about prefixing the config options with
> `sql-driver` instead of just `sql` to be more specific?
>
> Best,
> Mason
>
> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
>
> <mailto:
>
> paullin3280@gmail.com>> wrote:
>
>
> Hi Jark,
>
> Thanks for your input! Please see my comments inline.
>
> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> DataStream API also doesn't provide a default main class for users,
> why do we need to provide such one for SQL?
>
>
> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>
> submitted
>
> via Flink CLI which actually could be DataStream/Table jobs.
>
> I think a default main class would be user-friendly which eliminates
>
> the
>
> need
> for users to write a main class as SQLRunner in Flink K8s operator
>
> [1].
>
>
> I thought the proposed SqlDriver was a dedicated main class
>
> accepting
>
> SQL files, is
>
> that correct?
>
>
> Both JSON plans and SQL files are accepted. SQL Gateway should use
>
> JSON
>
> plans,
> while CLI users may use either JSON plans or SQL files.
>
> Please see the updated FLIP[2] for more details.
>
> Personally, I prefer the way of init containers which doesn't
>
> depend
>
> on
>
> additional components.
> This can reduce the moving parts of a production environment.
> Depending on a distributed file system makes the testing, demo, and
>
> local
>
> setup harder than init containers.
>
>
> Please note that we could reuse the checkpoint storage like S3/HDFS,
>
> which
>
> should
> be required to run Flink in production, so I guess that would be
> acceptable for most
> users. WDYT?
>
> WRT testing, demo, and local setups, I think we could support the
>
> local
>
> filesystem
> scheme i.e. file://** as the state backends do. It works as long as
>
> SQL
>
> Gateway
> and JobManager(or SQL Driver) can access the resource directory
>
> (specified
>
> via
> `sql-gateway.application.storage-dir`).
>
> Thanks!
>
> [1]
>
>
>
>
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
>
> [2]
>
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>
> [3]
>
>
>
>
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>
>
> Best,
> Paul Lam
>
> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
>
> Hi Paul,
>
> Thanks for your reply. I left my comments inline.
>
> As the FLIP said, it’s good to have a default main class for Flink
>
> SQLs,
>
> which allows users to submit Flink SQLs in the same way as
>
> DataStream
>
> jobs, or else users need to write their own main class.
>
>
> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> DataStream API also doesn't provide a default main class for users,
> why do we need to provide such one for SQL?
>
> With the help of ExecNodeGraph, do we still need the serialized
> SessionState? If not, we could make SQL Driver accepts two
>
> serialized
>
> formats:
>
>
> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
>
> the
>
> proposed SqlDriver was a dedicated main class accepting SQL files,
>
> is
>
> that correct?
> If true, we have to ship the SessionState for this case which is a
>
> large
>
> work.
> I think we just need a JsonPlanDriver which is a main class that
>
> accepts
>
> JsonPlan as the parameter.
>
>
> The common solutions I know is to use distributed file systems or
>
> use
>
> init containers to localize the resources.
>
>
> Personally, I prefer the way of init containers which doesn't
>
> depend
>
> on
>
> additional components.
> This can reduce the moving parts of a production environment.
> Depending on a distributed file system makes the testing, demo, and
>
> local
>
> setup harder than init containers.
>
> Best,
> Jark
>
>
>
>
> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
>
> <mailto:
>
> paullin3280@gmail.com> <mailto:
>
> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>
>
> The FLIP is in the early phase and some details are not included,
>
> but
>
> fortunately, we got lots of valuable ideas from the discussion.
>
> Thanks to everyone who joined the dissuasion!
> @Weihua @Shanmon @Shengkai @Biao @Jark
>
> This weekend I’m gonna revisit and update the FLIP, adding more
> details. Hopefully, we can further align our opinions.
>
> Best,
> Paul Lam
>
> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
>
> paullin3280@gmail.com>> 写道：
>
>
> Hi Jark,
>
> Thanks a lot for your input!
>
> If we decide to submit ExecNodeGraph instead of SQL file, is it
>
> still
>
> necessary to support SQL Driver?
>
>
> I think so. Apart from usage in SQL Gateway, SQL Driver could
>
> simplify
>
> Flink SQL execution with Flink CLI.
>
> As the FLIP said, it’s good to have a default main class for
>
> Flink
>
> SQLs,
>
> which allows users to submit Flink SQLs in the same way as
>
> DataStream
>
> jobs, or else users need to write their own main class.
>
> SQL Driver needs to serialize SessionState which is very
>
> challenging
>
> but not detailed covered in the FLIP.
>
>
> With the help of ExecNodeGraph, do we still need the serialized
> SessionState? If not, we could make SQL Driver accepts two
>
> serialized
>
> formats:
>
> - SQL files for user-facing public usage
> - ExecNodeGraph for internal usage
>
> It’s kind of similar to the relationship between job jars and
>
> jobgraphs.
>
>
> Regarding "K8S doesn't support shipping multiple jars", is that
>
> true?
>
> Is it
>
> possible to support it?
>
>
> Yes, K8s doesn’t distribute any files. It’s the users’
>
> responsibility
>
> to
>
> make
>
> sure the resources are accessible in the containers. The common
>
> solutions
>
> I know is to use distributed file systems or use init containers
>
> to
>
> localize the
>
> resources.
>
> Now I lean toward introducing a fs to do the distribution job.
>
> WDYT?
>
>
> Best,
> Paul Lam
>
> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
>
> imjark@gmail.com>
>
> <mailto:imjark@gmail.com <ma...@gmail.com>>
>
> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
>
> imjark@gmail.com <ma...@gmail.com>>>>
>
> 写道：
>
>
> Hi Paul,
>
> Thanks for starting this discussion. I like the proposal! This
>
> is
>
> a
>
> frequently requested feature!
>
> I agree with Shengkai that ExecNodeGraph as the submission
>
> object
>
> is a
>
> better idea than SQL file. To be more specific, it should be
>
> JsonPlanGraph
>
> or CompiledPlan which is the serializable representation.
>
> CompiledPlan
>
> is a
>
> clear separation between compiling/optimization/validation and
>
> execution.
>
> This can keep the validation and metadata accessing still on the
>
> SQLGateway
>
> side. This allows SQLGateway to leverage some metadata caching
>
> and
>
> UDF
>
> JAR
>
> caching for better compiling performance.
>
> If we decide to submit ExecNodeGraph instead of SQL file, is it
>
> still
>
> necessary to support SQL Driver? Regarding non-interactive SQL
>
> jobs,
>
> users
>
> can use the Table API program for application mode. SQL Driver
>
> needs
>
> to
>
> serialize SessionState which is very challenging but not
>
> detailed
>
> covered
>
> in the FLIP.
>
> Regarding "K8S doesn't support shipping multiple jars", is that
>
> true?
>
> Is it
>
> possible to support it?
>
> Best,
> Jark
>
>
>
> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
>
> <ma...@gmail.com> <mailto:
>
> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>
> paullin3280@gmail.com <ma...@gmail.com> <mailto:
>
> paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>
>
> Hi Weihua,
>
> You’re right. Distributing the SQLs to the TMs is one of the
>
> challenging
>
> parts of this FLIP.
>
> Web submission is not enabled in application mode currently as
>
> you
>
> said,
>
> but it could be changed if we have good reasons.
>
> What do you think about introducing a distributed storage for
>
> SQL
>
> Gateway?
>
>
> We could make use of Flink file systems [1] to distribute the
>
> SQL
>
> Gateway
>
> generated resources, that should solve the problem at its root
>
> cause.
>
>
> Users could specify Flink-supported file systems to ship files.
>
> It’s
>
> only
>
> required when using SQL Gateway with K8s application mode.
>
> [1]
>
>
>
>
>
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
> <
>
>
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
>
> <
>
>
>
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
> <
>
>
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
>
>
> <
>
>
>
>
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
> <
>
>
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
>
> <
>
>
>
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
> <
>
>
>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
>
>
>
>
> Best,
> Paul Lam
>
> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
>
> huweihua.ckl@gmail.com> <mailto:
>
> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
>
>
> Thanks Paul for your reply.
>
> SQLDriver looks good to me.
>
> 2. Do you mean a pass the SQL string a configuration or a
>
> program
>
> argument?
>
>
>
> I brought this up because we were unable to pass the SQL file
>
> to
>
> Flink
>
> using Kubernetes mode.
> For DataStream/Python users, they need to prepare their images
>
> for
>
> the
>
> jars
>
> and dependencies.
> But for SQL users, they can use a common image to run
>
> different
>
> SQL
>
> queries
>
> if there are no other udf requirements.
> It would be great if the SQL query and image were not bound.
>
> Using strings is a way to decouple these, but just as you
>
> mentioned,
>
> it's
>
> not easy to pass complex SQL.
>
> use web submission
>
> AFAIK, we can not use web submission in the Application mode.
>
> Please
>
> correct me if I'm wrong.
>
>
> Best,
> Weihua
>
>
> On Wed, May 31, 2023 at 9:37 PM Paul Lam <
>
> paullin3280@gmail.com
>
> <ma...@gmail.com>
>
> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>
> wrote:
>
>
> Hi Biao,
>
> Thanks for your comments!
>
> 1. Scope: is this FLIP only targeted for non-interactive
>
> Flink
>
> SQL
>
> jobs
>
> in
>
> Application mode? More specifically, if we use SQL
>
> client/gateway
>
> to
>
> execute some interactive SQLs like a SELECT query, can we
>
> ask
>
> flink
>
> to
>
> use
>
> Application mode to execute those queries after this FLIP?
>
>
> Thanks for pointing it out. I think only DMLs would be
>
> executed
>
> via
>
> SQL
>
> Driver.
> I'll add the scope to the FLIP.
>
> 2. Deployment: I believe in YARN mode, the implementation is
>
> trivial as
>
> we
>
> can ship files via YARN's tool easily but for K8s, things
>
> can
>
> be
>
> more
>
> complicated as Shengkai said.
>
>
>
> Your input is very informative. I’m thinking about using web
>
> submission,
>
> but it requires exposing the JobManager port which could also
>
> be
>
> a
>
> problem
>
> on K8s.
>
> Another approach is to explicitly require a distributed
>
> storage
>
> to
>
> ship
>
> files,
> but we may need a new deployment executor for that.
>
> What do you think of these two approaches?
>
> 3. Serialization of SessionState: in SessionState, there are
>
> some
>
> unserializable fields
> like
>
> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>
> It
>
> may be worthwhile to add more details about the
>
> serialization
>
> part.
>
>
> I agree. That’s a missing part. But if we use ExecNodeGraph
>
> as
>
> Shengkai
>
> mentioned, do we eliminate the need for serialization of
>
> SessionState?
>
>
> Best,
> Paul Lam
>
> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
>
> biaogeng7@gmail.com> <mailto:
>
> biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
>
>
> Thanks Paul for the proposal!I believe it would be very
>
> useful
>
> for
>
> flink
>
> users.
> After reading the FLIP, I have some questions:
> 1. Scope: is this FLIP only targeted for non-interactive
>
> Flink
>
> SQL
>
> jobs
>
> in
>
> Application mode? More specifically, if we use SQL
>
> client/gateway
>
> to
>
> execute some interactive SQLs like a SELECT query, can we
>
> ask
>
> flink
>
> to
>
> use
>
> Application mode to execute those queries after this FLIP?
> 2. Deployment: I believe in YARN mode, the implementation is
>
> trivial as
>
> we
>
> can ship files via YARN's tool easily but for K8s, things
>
> can
>
> be
>
> more
>
> complicated as Shengkai said. I have implemented a simple
>
> POC
>
> <
>
>
>
>
>
>
>
>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>
> <
>
>
>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>
>
> <
>
>
>
>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>
> <
>
>
>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>
>
>
>
> based on SQL client before(i.e. consider the SQL client
>
> which
>
> supports
>
> executing a SQL file as the SQL driver in this FLIP). One
>
> problem
>
> I
>
> have
>
> met is how do we ship SQL files ( or Job Graph) to the k8s
>
> side.
>
> Without
>
> such support, users have to modify the initContainer or
>
> rebuild
>
> a
>
> new
>
> K8s
>
> image every time to fetch the SQL file. Like the flink k8s
>
> operator,
>
> one
>
> workaround is to utilize the flink config(transforming the
>
> SQL
>
> file
>
> to
>
> a
>
> escaped string like Weihua mentioned) which will be
>
> converted
>
> to a
>
> ConfigMap but K8s has size limit of ConfigMaps(no larger
>
> than
>
> 1MB
>
> <
>
> https://kubernetes.io/docs/concepts/configuration/configmap/
>
> <
>
> https://kubernetes.io/docs/concepts/configuration/configmap/> <
>
> https://kubernetes.io/docs/concepts/configuration/configmap/ <
>
> https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
>
> Not
>
> sure
>
> if we have better solutions.
> 3. Serialization of SessionState: in SessionState, there are
>
> some
>
> unserializable fields
> like
>
> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>
> It
>
> may be worthwhile to add more details about the
>
> serialization
>
> part.
>
>
> Best,
> Biao Geng
>
> Paul Lam <paullin3280@gmail.com <mailto:
>
> paullin3280@gmail.com
>
>
> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>
> 于2023年5月31日周三 11:49写道：
>
>
> Hi Weihua,
>
> Thanks a lot for your input! Please see my comments inline.
>
> - Is SQLRunner the better name? We use this to run a SQL
>
> Job.
>
> (Not
>
> strong,
>
> the SQLDriver is fine for me)
>
>
> I’ve thought about SQL Runner but picked SQL Driver for the
>
> following
>
> reasons FYI:
>
> 1. I have a PythonDriver doing the same job for PyFlink [1]
> 2. Flink program's main class is sort of like Driver in
>
> JDBC
>
> which
>
> translates SQLs into
> databases specific languages.
>
> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>
> - Could we run SQL jobs using SQL in strings? Otherwise,
>
> we
>
> need
>
> to
>
> prepare
>
> a SQL file in an image for Kubernetes application mode,
>
> which
>
> may
>
> be
>
> a
>
> bit
>
> cumbersome.
>
>
> Do you mean a pass the SQL string a configuration or a
>
> program
>
> argument?
>
>
> I thought it might be convenient for testing propose, but
>
> not
>
> recommended
>
> for production,
> cause Flink SQLs could be complicated and involves lots of
>
> characters
>
> that
>
> need to escape.
>
> WDYT?
>
> - I noticed that we don't specify the SQLDriver jar in the
>
> "run-application"
>
> command. Does that mean we need to perform automatic
>
> detection
>
> in
>
> Flink?
>
>
> Yes! It’s like running a PyFlink job with the following
>
> command:
>
>
> ```
> ./bin/flink run \
> --pyModule table.word_count \
> --pyFiles examples/python/table
> ```
>
> The CLI determines if it’s a SQL job, if yes apply the SQL
>
> Driver
>
> automatically.
>
>
> [1]
>
>
>
>
>
>
>
>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>
> <
>
>
>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>
>
> <
>
>
>
>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>
> <
>
>
>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>
>
>
>
> Best,
> Paul Lam
>
> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com
>
> <mailto:
>
> huweihua.ckl@gmail.com> <mailto:
>
> huweihua.ckl@gmail.com>> 写道：
>
>
> Thanks Paul for the proposal.
>
> +1 for this. It is valuable in improving ease of use.
>
> I have a few questions.
> - Is SQLRunner the better name? We use this to run a SQL
>
> Job.
>
> (Not
>
> strong,
>
> the SQLDriver is fine for me)
> - Could we run SQL jobs using SQL in strings? Otherwise,
>
> we
>
> need
>
> to
>
> prepare
>
> a SQL file in an image for Kubernetes application mode,
>
> which
>
> may
>
> be
>
> a
>
> bit
>
> cumbersome.
> - I noticed that we don't specify the SQLDriver jar in the
>
> "run-application"
>
> command. Does that mean we need to perform automatic
>
> detection
>
> in
>
> Flink?
>
>
>
> Best,
> Weihua
>
>
> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>
> paullin3280@gmail.com
>
> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>
> wrote:
>
>
> Hi team,
>
> I’d like to start a discussion about FLIP-316 [1], which
>
> introduces
>
> a
>
> SQL
>
> driver as the
> default main class for Flink SQL jobs.
>
> Currently, Flink SQL could be executed out of the box
>
> either
>
> via
>
> SQL
>
> Client/Gateway
> or embedded in a Flink Java/Python program.
>
> However, each one has its drawback:
>
> - SQL Client/Gateway doesn’t support the application
>
> deployment
>
> mode
>
> [2]
>
> - Flink Java/Python program requires extra work to write
>
> a
>
> non-SQL
>
> program
>
>
> Therefore, I propose adding a SQL driver to act as the
>
> default
>
> main
>
> class
>
> for SQL jobs.
> Please see the FLIP docs for details and feel free to
>
> comment.
>
> Thanks!
>
>
> [1]
>
>
>
>
>
>
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>
> <
>
>
>
>
>
>
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>
>
> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> https://issues.apache.org/jira/browse/FLINK-26541>
>
> Best,
> Paul Lam
>
>
>
>
>
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Mason,
I get your point. I'm increasingly feeling the need to introduce a built-in
file distribution mechanism for flink-kubernetes module, just like Spark
does with `spark.kubernetes.file.upload.path` [1].
I’m assuming the workflow is as follows:
- KubernetesClusterDescripter uploads all local resources to a remote
storage via Flink filesystem (skips if the resources are already remote).
- KubernetesApplicationClusterEntrypoint downloads the resources
and put them in the classpath during startup.
I wouldn't mind splitting it into another FLIP to ensure that everything is
done correctly.
cc'ed @Yang to gather more opinions.
[1] https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management
Best,
Paul Lam
> 2023年6月8日 12:15，Mason Chen <ma...@gmail.com> 写道：
>
> Hi Paul,
>
> Thanks for your response!
>
> I agree that utilizing SQL Drivers in Java applications is equally important
>> as employing them in SQL Gateway. WRT init containers, I think most
>> users use them just as a workaround. For example, wget a jar from the
>> maven repo.
>>
>> We could implement the functionality in SQL Driver in a more graceful
>> way and the flink-supported filesystem approach seems to be a
>> good choice.
>>
>
> My main point is: can we solve the problem with a design agnostic of SQL
> and Stream API? I mentioned a use case where this ability is useful for
> Java or Stream API applications. Maybe this is even a non-goal to your FLIP
> since you are focusing on the driver entrypoint.
>
> Jark mentioned some optimizations:
>
>> This allows SQLGateway to leverage some metadata caching and UDF JAR
>> caching for better compiling performance.
>>
> It would be great to see this even outside the SQLGateway (i.e. UDF JAR
> caching).
>
> Best,
> Mason
>
> On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fs...@gmail.com> wrote:
>
>> Hi. Paul. Thanks for your update and the update makes me understand the
>> design much better.
>>
>> But I still have some questions about the FLIP.
>>
>>> For SQL Gateway, only DMLs need to be delegated to the SQL server
>>> Driver. I would think about the details and update the FLIP. Do you have
>> some
>>> ideas already?
>>
>> If the applicaiton mode can not support library mode, I think we should
>> only execute INSERT INTO and UPDATE/ DELETE statement in the application
>> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
>> statements. The ANALYZE TABLE syntax need to register the statistic to the
>> catalog after job finishes and the CALL PROCEDURE statement doesn't
>> generate the ExecNodeGraph.
>>
>> * Introduce storage via option `sql-gateway.application.storage-dir`
>>
>> If we can not support to submit the jars through web submission, +1 to
>> introduce the options to upload the files. While I think the uploader
>> should be responsible to remove the uploaded jars. Can we remove the jars
>> if the job is running or gateway exits?
>>
>> * JobID is not avaliable
>>
>> Can we use the returned rest client by ApplicationDeployer to query the job
>> id? I am concerned that users don't know which job is related to the
>> submitted SQL.
>>
>> * Do we need to introduce a new module named flink-table-sql-runner?
>>
>> It seems we need to introduce a new module. Will the new module is
>> available in the distribution package? I agree with Jark that we don't need
>> to introduce this for table-API users and these users have their main
>> class. If we want to make users write the k8s operator more easily, I think
>> we should modify the k8s operator repo. If we don't need to support SQL
>> files, can we make this jar only visible in the sql-gateway like we do in
>> the planner loader?[1]
>>
>> [1]
>>
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
>>
>> Best,
>> Shengkai
>>
>>
>>
>>
>>
>>
>>
>>
>> Weihua Hu <hu...@gmail.com> 于2023年6月7日周三 10:52写道：
>>
>>> Hi,
>>>
>>> Thanks for updating the FLIP.
>>>
>>> I have two cents on the distribution of SQLs and resources.
>>> 1. Should we support a common file distribution mechanism for k8s
>>> application mode?
>>> I have seen some issues and requirements on the mailing list.
>>> In our production environment, we implement the download command in the
>>> CliFrontend.
>>> And automatically add an init container to the POD for file
>> downloading.
>>> The advantage of this
>>> is that we can use all Flink-supported file systems to store files.
>>>
>>> This need more discussion. I would appreciate hearing more opinions.
>>>
>>> 2. In this FLIP, we distribute files in two different ways in YARN and
>>> Kubernetes. Can we combine it in one way?
>>> If we don't want to implement a common file distribution for k8s
>>> application mode. Could we use the SQLDriver
>>> to download the files both in YARN and K8S? IMO, this can reduce the
>> cost
>>> of code maintenance.
>>>
>>> Best,
>>> Weihua
>>>
>>>
>>> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
>>>
>>>> Hi Mason,
>>>>
>>>> Thanks for your input!
>>>>
>>>>> +1 for init containers or a more generalized way of obtaining
>> arbitrary
>>>>> files. File fetching isn't specific to just SQL--it also matters for
>>> Java
>>>>> applications if the user doesn't want to rebuild a Flink image and
>> just
>>>>> wants to modify the user application fat jar.
>>>>
>>>> I agree that utilizing SQL Drivers in Java applications is equally
>>>> important
>>>> as employing them in SQL Gateway. WRT init containers, I think most
>>>> users use them just as a workaround. For example, wget a jar from the
>>>> maven repo.
>>>>
>>>> We could implement the functionality in SQL Driver in a more graceful
>>>> way and the flink-supported filesystem approach seems to be a
>>>> good choice.
>>>>
>>>>> Also, what do you think about prefixing the config options with
>>>>> `sql-driver` instead of just `sql` to be more specific?
>>>>
>>>> LGTM, since SQL Driver is a public interface and the options are
>>>> specific to it.
>>>>
>>>> Best,
>>>> Paul Lam
>>>>
>>>>> 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> +1 for this feature and supporting SQL file + JSON plans. We get a
>> lot
>>> of
>>>>> requests to just be able to submit a SQL file, but the JSON plan
>>>>> optimizations make sense.
>>>>>
>>>>> +1 for init containers or a more generalized way of obtaining
>> arbitrary
>>>>> files. File fetching isn't specific to just SQL--it also matters for
>>> Java
>>>>> applications if the user doesn't want to rebuild a Flink image and
>> just
>>>>> wants to modify the user application fat jar.
>>>>>
>>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>>> which
>>>>>> should
>>>>>
>>>>> be required to run Flink in production, so I guess that would be
>>>> acceptable
>>>>>> for most
>>>>>
>>>>> users. WDYT?
>>>>>
>>>>>
>>>>> If you do go this route, it would be nice to support writing these
>>> files
>>>> to
>>>>> S3/HDFS via Flink. This makes access control and policy management
>>>> simpler.
>>>>>
>>>>> Also, what do you think about prefixing the config options with
>>>>> `sql-driver` instead of just `sql` to be more specific?
>>>>>
>>>>> Best,
>>>>> Mason
>>>>>
>>>>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
>>> <mailto:
>>>> paullin3280@gmail.com>> wrote:
>>>>>
>>>>>> Hi Jark,
>>>>>>
>>>>>> Thanks for your input! Please see my comments inline.
>>>>>>
>>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>>>> DataStream API also doesn't provide a default main class for users,
>>>>>>> why do we need to provide such one for SQL?
>>>>>>
>>>>>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>>>> submitted
>>>>>> via Flink CLI which actually could be DataStream/Table jobs.
>>>>>>
>>>>>> I think a default main class would be user-friendly which eliminates
>>> the
>>>>>> need
>>>>>> for users to write a main class as SQLRunner in Flink K8s operator
>>> [1].
>>>>>>
>>>>>>> I thought the proposed SqlDriver was a dedicated main class
>> accepting
>>>>>> SQL files, is
>>>>>>> that correct?
>>>>>>
>>>>>> Both JSON plans and SQL files are accepted. SQL Gateway should use
>>> JSON
>>>>>> plans,
>>>>>> while CLI users may use either JSON plans or SQL files.
>>>>>>
>>>>>> Please see the updated FLIP[2] for more details.
>>>>>>
>>>>>>> Personally, I prefer the way of init containers which doesn't
>> depend
>>> on
>>>>>>> additional components.
>>>>>>> This can reduce the moving parts of a production environment.
>>>>>>> Depending on a distributed file system makes the testing, demo, and
>>>> local
>>>>>>> setup harder than init containers.
>>>>>>
>>>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>>>> which
>>>>>> should
>>>>>> be required to run Flink in production, so I guess that would be
>>>>>> acceptable for most
>>>>>> users. WDYT?
>>>>>>
>>>>>> WRT testing, demo, and local setups, I think we could support the
>>> local
>>>>>> filesystem
>>>>>> scheme i.e. file://** as the state backends do. It works as long as
>>> SQL
>>>>>> Gateway
>>>>>> and JobManager(or SQL Driver) can access the resource directory
>>>> (specified
>>>>>> via
>>>>>> `sql-gateway.application.storage-dir`).
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> [1]
>>>>>>
>>>>
>>>
>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
>>>>>> [2]
>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>> [3]
>>>>>>
>>>>
>>>
>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>>>>>>
>>>>>> Best,
>>>>>> Paul Lam
>>>>>>
>>>>>>> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
>>>>>>>
>>>>>>> Hi Paul,
>>>>>>>
>>>>>>> Thanks for your reply. I left my comments inline.
>>>>>>>
>>>>>>>> As the FLIP said, it’s good to have a default main class for Flink
>>>> SQLs,
>>>>>>>> which allows users to submit Flink SQLs in the same way as
>>> DataStream
>>>>>>>> jobs, or else users need to write their own main class.
>>>>>>>
>>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>>>> DataStream API also doesn't provide a default main class for users,
>>>>>>> why do we need to provide such one for SQL?
>>>>>>>
>>>>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>>>>> SessionState? If not, we could make SQL Driver accepts two
>>> serialized
>>>>>>>> formats:
>>>>>>>
>>>>>>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
>>> the
>>>>>>> proposed SqlDriver was a dedicated main class accepting SQL files,
>> is
>>>>>>> that correct?
>>>>>>> If true, we have to ship the SessionState for this case which is a
>>>> large
>>>>>>> work.
>>>>>>> I think we just need a JsonPlanDriver which is a main class that
>>>> accepts
>>>>>>> JsonPlan as the parameter.
>>>>>>>
>>>>>>>
>>>>>>>> The common solutions I know is to use distributed file systems or
>>> use
>>>>>>>> init containers to localize the resources.
>>>>>>>
>>>>>>> Personally, I prefer the way of init containers which doesn't
>> depend
>>> on
>>>>>>> additional components.
>>>>>>> This can reduce the moving parts of a production environment.
>>>>>>> Depending on a distributed file system makes the testing, demo, and
>>>> local
>>>>>>> setup harder than init containers.
>>>>>>>
>>>>>>> Best,
>>>>>>> Jark
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
>>> <mailto:
>>>> paullin3280@gmail.com> <mailto:
>>>>>> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>>>>>>>
>>>>>>>> The FLIP is in the early phase and some details are not included,
>>> but
>>>>>>>> fortunately, we got lots of valuable ideas from the discussion.
>>>>>>>>
>>>>>>>> Thanks to everyone who joined the dissuasion!
>>>>>>>> @Weihua @Shanmon @Shengkai @Biao @Jark
>>>>>>>>
>>>>>>>> This weekend I’m gonna revisit and update the FLIP, adding more
>>>>>>>> details. Hopefully, we can further align our opinions.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Paul Lam
>>>>>>>>
>>>>>>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
>>>> paullin3280@gmail.com>> 写道：
>>>>>>>>>
>>>>>>>>> Hi Jark,
>>>>>>>>>
>>>>>>>>> Thanks a lot for your input!
>>>>>>>>>
>>>>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>>> still
>>>>>>>>>> necessary to support SQL Driver?
>>>>>>>>>
>>>>>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
>>>> simplify
>>>>>>>>> Flink SQL execution with Flink CLI.
>>>>>>>>>
>>>>>>>>> As the FLIP said, it’s good to have a default main class for
>> Flink
>>>>>> SQLs,
>>>>>>>>> which allows users to submit Flink SQLs in the same way as
>>> DataStream
>>>>>>>>> jobs, or else users need to write their own main class.
>>>>>>>>>
>>>>>>>>>> SQL Driver needs to serialize SessionState which is very
>>> challenging
>>>>>>>>>> but not detailed covered in the FLIP.
>>>>>>>>>
>>>>>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>>>>>> SessionState? If not, we could make SQL Driver accepts two
>>> serialized
>>>>>>>>> formats:
>>>>>>>>>
>>>>>>>>> - SQL files for user-facing public usage
>>>>>>>>> - ExecNodeGraph for internal usage
>>>>>>>>>
>>>>>>>>> It’s kind of similar to the relationship between job jars and
>>>>>> jobgraphs.
>>>>>>>>>
>>>>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>>>> true?
>>>>>>>> Is it
>>>>>>>>>> possible to support it?
>>>>>>>>>
>>>>>>>>> Yes, K8s doesn’t distribute any files. It’s the users’
>>> responsibility
>>>>>> to
>>>>>>>> make
>>>>>>>>> sure the resources are accessible in the containers. The common
>>>>>> solutions
>>>>>>>>> I know is to use distributed file systems or use init containers
>> to
>>>>>>>> localize the
>>>>>>>>> resources.
>>>>>>>>>
>>>>>>>>> Now I lean toward introducing a fs to do the distribution job.
>>> WDYT?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Paul Lam
>>>>>>>>>
>>>>>>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
>>> imjark@gmail.com>
>>>> <mailto:imjark@gmail.com <ma...@gmail.com>>
>>>>>> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
>>>> imjark@gmail.com <ma...@gmail.com>>>>
>>>>>>>> 写道：
>>>>>>>>>>
>>>>>>>>>> Hi Paul,
>>>>>>>>>>
>>>>>>>>>> Thanks for starting this discussion. I like the proposal! This
>> is
>>> a
>>>>>>>>>> frequently requested feature!
>>>>>>>>>>
>>>>>>>>>> I agree with Shengkai that ExecNodeGraph as the submission
>> object
>>>> is a
>>>>>>>>>> better idea than SQL file. To be more specific, it should be
>>>>>>>> JsonPlanGraph
>>>>>>>>>> or CompiledPlan which is the serializable representation.
>>>> CompiledPlan
>>>>>>>> is a
>>>>>>>>>> clear separation between compiling/optimization/validation and
>>>>>>>> execution.
>>>>>>>>>> This can keep the validation and metadata accessing still on the
>>>>>>>> SQLGateway
>>>>>>>>>> side. This allows SQLGateway to leverage some metadata caching
>> and
>>>> UDF
>>>>>>>> JAR
>>>>>>>>>> caching for better compiling performance.
>>>>>>>>>>
>>>>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>>>> still
>>>>>>>>>> necessary to support SQL Driver? Regarding non-interactive SQL
>>> jobs,
>>>>>>>> users
>>>>>>>>>> can use the Table API program for application mode. SQL Driver
>>> needs
>>>>>> to
>>>>>>>>>> serialize SessionState which is very challenging but not
>> detailed
>>>>>>>> covered
>>>>>>>>>> in the FLIP.
>>>>>>>>>>
>>>>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>>>> true?
>>>>>>>> Is it
>>>>>>>>>> possible to support it?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Jark
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
>>>> <ma...@gmail.com> <mailto:
>>>>>> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>>>>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:
>>>> paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Weihua,
>>>>>>>>>>>
>>>>>>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
>>>>>>>> challenging
>>>>>>>>>>> parts of this FLIP.
>>>>>>>>>>>
>>>>>>>>>>> Web submission is not enabled in application mode currently as
>>> you
>>>>>>>> said,
>>>>>>>>>>> but it could be changed if we have good reasons.
>>>>>>>>>>>
>>>>>>>>>>> What do you think about introducing a distributed storage for
>> SQL
>>>>>>>> Gateway?
>>>>>>>>>>>
>>>>>>>>>>> We could make use of Flink file systems [1] to distribute the
>> SQL
>>>>>>>> Gateway
>>>>>>>>>>> generated resources, that should solve the problem at its root
>>>> cause.
>>>>>>>>>>>
>>>>>>>>>>> Users could specify Flink-supported file systems to ship files.
>>>> It’s
>>>>>>>> only
>>>>>>>>>>> required when using SQL Gateway with K8s application mode.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>>
>>>>>> <
>>>>>>
>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>>
>>>>>>>
>>>>>>>> <
>>>>>>>>
>>>>>>
>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>>
>>>>>> <
>>>>>>
>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>> <
>>>>
>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Paul Lam
>>>>>>>>>>>
>>>>>>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
>>>> huweihua.ckl@gmail.com> <mailto:
>>>>>> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Paul for your reply.
>>>>>>>>>>>>
>>>>>>>>>>>> SQLDriver looks good to me.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Do you mean a pass the SQL string a configuration or a
>>> program
>>>>>>>>>>> argument?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I brought this up because we were unable to pass the SQL file
>> to
>>>>>> Flink
>>>>>>>>>>>> using Kubernetes mode.
>>>>>>>>>>>> For DataStream/Python users, they need to prepare their images
>>> for
>>>>>> the
>>>>>>>>>>> jars
>>>>>>>>>>>> and dependencies.
>>>>>>>>>>>> But for SQL users, they can use a common image to run
>> different
>>>> SQL
>>>>>>>>>>> queries
>>>>>>>>>>>> if there are no other udf requirements.
>>>>>>>>>>>> It would be great if the SQL query and image were not bound.
>>>>>>>>>>>>
>>>>>>>>>>>> Using strings is a way to decouple these, but just as you
>>>> mentioned,
>>>>>>>> it's
>>>>>>>>>>>> not easy to pass complex SQL.
>>>>>>>>>>>>
>>>>>>>>>>>>> use web submission
>>>>>>>>>>>> AFAIK, we can not use web submission in the Application mode.
>>>> Please
>>>>>>>>>>>> correct me if I'm wrong.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Weihua
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <
>> paullin3280@gmail.com
>>>> <ma...@gmail.com>
>>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Biao,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your comments!
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
>> Flink
>>>> SQL
>>>>>>>> jobs
>>>>>>>>>>>>> in
>>>>>>>>>>>>>> Application mode? More specifically, if we use SQL
>>>> client/gateway
>>>>>> to
>>>>>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we
>> ask
>>>>>> flink
>>>>>>>> to
>>>>>>>>>>>>> use
>>>>>>>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for pointing it out. I think only DMLs would be
>> executed
>>>> via
>>>>>>>> SQL
>>>>>>>>>>>>> Driver.
>>>>>>>>>>>>> I'll add the scope to the FLIP.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>>>> trivial as
>>>>>>>>>>>>> we
>>>>>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things
>> can
>>> be
>>>>>>>> more
>>>>>>>>>>>>>> complicated as Shengkai said.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your input is very informative. I’m thinking about using web
>>>>>>>> submission,
>>>>>>>>>>>>> but it requires exposing the JobManager port which could also
>>> be
>>>> a
>>>>>>>>>>> problem
>>>>>>>>>>>>> on K8s.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another approach is to explicitly require a distributed
>> storage
>>>> to
>>>>>>>> ship
>>>>>>>>>>>>> files,
>>>>>>>>>>>>> but we may need a new deployment executor for that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you think of these two approaches?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
>>>> some
>>>>>>>>>>>>>> unserializable fields
>>>>>>>>>>>>>> like
>>>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>>>>>> It
>>>>>>>>>>>>>> may be worthwhile to add more details about the
>> serialization
>>>>>> part.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph
>> as
>>>>>>>> Shengkai
>>>>>>>>>>>>> mentioned, do we eliminate the need for serialization of
>>>>>>>> SessionState?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Paul Lam
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
>>>> biaogeng7@gmail.com> <mailto:
>>>>>> biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Paul for the proposal!I believe it would be very
>> useful
>>>> for
>>>>>>>>>>> flink
>>>>>>>>>>>>>> users.
>>>>>>>>>>>>>> After reading the FLIP, I have some questions:
>>>>>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
>> Flink
>>>> SQL
>>>>>>>> jobs
>>>>>>>>>>>>> in
>>>>>>>>>>>>>> Application mode? More specifically, if we use SQL
>>>> client/gateway
>>>>>> to
>>>>>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we
>> ask
>>>>>> flink
>>>>>>>> to
>>>>>>>>>>>>> use
>>>>>>>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>>>> trivial as
>>>>>>>>>>>>> we
>>>>>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things
>> can
>>> be
>>>>>>>> more
>>>>>>>>>>>>>> complicated as Shengkai said. I have implemented a simple
>> POC
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>> <
>>>>
>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>>
>>>>>> <
>>>>>>
>>>>
>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>> <
>>>>
>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>>
>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> based on SQL client before(i.e. consider the SQL client
>> which
>>>>>>>> supports
>>>>>>>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One
>>>> problem
>>>>>> I
>>>>>>>>>>> have
>>>>>>>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s
>>> side.
>>>>>>>>>>> Without
>>>>>>>>>>>>>> such support, users have to modify the initContainer or
>>> rebuild
>>>> a
>>>>>>>> new
>>>>>>>>>>> K8s
>>>>>>>>>>>>>> image every time to fetch the SQL file. Like the flink k8s
>>>>>> operator,
>>>>>>>>>>> one
>>>>>>>>>>>>>> workaround is to utilize the flink config(transforming the
>> SQL
>>>>>> file
>>>>>>>> to
>>>>>>>>>>> a
>>>>>>>>>>>>>> escaped string like Weihua mentioned) which will be
>> converted
>>>> to a
>>>>>>>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger
>> than
>>>> 1MB
>>>>>>>>>>>>>> <
>> https://kubernetes.io/docs/concepts/configuration/configmap/
>>> <
>>>> https://kubernetes.io/docs/concepts/configuration/configmap/> <
>>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ <
>>>> https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
>>>>>>>> Not
>>>>>>>>>>>>> sure
>>>>>>>>>>>>>> if we have better solutions.
>>>>>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
>>>> some
>>>>>>>>>>>>>> unserializable fields
>>>>>>>>>>>>>> like
>>>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>>>>>> It
>>>>>>>>>>>>>> may be worthwhile to add more details about the
>> serialization
>>>>>> part.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Biao Geng
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Paul Lam <paullin3280@gmail.com <mailto:
>> paullin3280@gmail.com
>>>>
>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>>>> 于2023年5月31日周三 11:49写道：
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Weihua,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
>>> Job.
>>>>>> (Not
>>>>>>>>>>>>>>> strong,
>>>>>>>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
>>>>>>>> following
>>>>>>>>>>>>>>> reasons FYI:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>>>>>>>>>>>> 2. Flink program's main class is sort of like Driver in
>> JDBC
>>>>>> which
>>>>>>>>>>>>>>> translates SQLs into
>>>>>>>>>>>>>>> databases specific languages.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
>> we
>>>> need
>>>>>>>> to
>>>>>>>>>>>>>>> prepare
>>>>>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode,
>>> which
>>>>>> may
>>>>>>>> be
>>>>>>>>>>> a
>>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>>> cumbersome.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do you mean a pass the SQL string a configuration or a
>>> program
>>>>>>>>>>> argument?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I thought it might be convenient for testing propose, but
>> not
>>>>>>>>>>>>> recommended
>>>>>>>>>>>>>>> for production,
>>>>>>>>>>>>>>> cause Flink SQLs could be complicated and involves lots of
>>>>>>>> characters
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> need to escape.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>>>>>>>> "run-application"
>>>>>>>>>>>>>>>> command. Does that mean we need to perform automatic
>>> detection
>>>>>> in
>>>>>>>>>>>>> Flink?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes! It’s like running a PyFlink job with the following
>>>> command:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>> ./bin/flink run \
>>>>>>>>>>>>>>> --pyModule table.word_count \
>>>>>>>>>>>>>>> --pyFiles examples/python/table
>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
>>>> Driver
>>>>>>>>>>>>>>> automatically.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>> <
>>>>
>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>>
>>>>>> <
>>>>>>
>>>>
>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>> <
>>>>
>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>>
>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Paul Lam
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com
>> <mailto:
>>>> huweihua.ckl@gmail.com> <mailto:
>>>>>> huweihua.ckl@gmail.com>> 写道：
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks Paul for the proposal.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have a few questions.
>>>>>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
>>> Job.
>>>>>> (Not
>>>>>>>>>>>>>>> strong,
>>>>>>>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
>> we
>>>> need
>>>>>>>> to
>>>>>>>>>>>>>>> prepare
>>>>>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode,
>>> which
>>>>>> may
>>>>>>>> be
>>>>>>>>>>> a
>>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>>> cumbersome.
>>>>>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>>>>>>>> "run-application"
>>>>>>>>>>>>>>>> command. Does that mean we need to perform automatic
>>> detection
>>>>>> in
>>>>>>>>>>>>> Flink?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Weihua
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>>>> paullin3280@gmail.com
>>>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi team,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
>>>>>>>> introduces
>>>>>>>>>>> a
>>>>>>>>>>>>>>> SQL
>>>>>>>>>>>>>>>>> driver as the
>>>>>>>>>>>>>>>>> default main class for Flink SQL jobs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Currently, Flink SQL could be executed out of the box
>>> either
>>>>>> via
>>>>>>>> SQL
>>>>>>>>>>>>>>>>> Client/Gateway
>>>>>>>>>>>>>>>>> or embedded in a Flink Java/Python program.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, each one has its drawback:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application
>>>> deployment
>>>>>>>> mode
>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>> - Flink Java/Python program requires extra work to write
>> a
>>>>>>>> non-SQL
>>>>>>>>>>>>>>> program
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the
>>>> default
>>>>>>>> main
>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>> for SQL jobs.
>>>>>>>>>>>>>>>>> Please see the FLIP docs for details and feel free to
>>>> comment.
>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Paul Lam
>>>>
>>>>
>>>
>>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Mason Chen <ma...@gmail.com>.
Hi Paul,
Thanks for your response!
I agree that utilizing SQL Drivers in Java applications is equally important
> as employing them in SQL Gateway. WRT init containers, I think most
> users use them just as a workaround. For example, wget a jar from the
> maven repo.
>
> We could implement the functionality in SQL Driver in a more graceful
> way and the flink-supported filesystem approach seems to be a
> good choice.
>
My main point is: can we solve the problem with a design agnostic of SQL
and Stream API? I mentioned a use case where this ability is useful for
Java or Stream API applications. Maybe this is even a non-goal to your FLIP
since you are focusing on the driver entrypoint.
Jark mentioned some optimizations:
> This allows SQLGateway to leverage some metadata caching and UDF JAR
> caching for better compiling performance.
>
It would be great to see this even outside the SQLGateway (i.e. UDF JAR
caching).
Best,
Mason
On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fs...@gmail.com> wrote:
> Hi. Paul. Thanks for your update and the update makes me understand the
> design much better.
>
> But I still have some questions about the FLIP.
>
> > For SQL Gateway, only DMLs need to be delegated to the SQL server
> > Driver. I would think about the details and update the FLIP. Do you have
> some
> > ideas already?
>
> If the applicaiton mode can not support library mode, I think we should
> only execute INSERT INTO and UPDATE/ DELETE statement in the application
> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
> statements. The ANALYZE TABLE syntax need to register the statistic to the
> catalog after job finishes and the CALL PROCEDURE statement doesn't
> generate the ExecNodeGraph.
>
> * Introduce storage via option `sql-gateway.application.storage-dir`
>
> If we can not support to submit the jars through web submission, +1 to
> introduce the options to upload the files. While I think the uploader
> should be responsible to remove the uploaded jars. Can we remove the jars
> if the job is running or gateway exits?
>
> * JobID is not avaliable
>
> Can we use the returned rest client by ApplicationDeployer to query the job
> id? I am concerned that users don't know which job is related to the
> submitted SQL.
>
> * Do we need to introduce a new module named flink-table-sql-runner?
>
> It seems we need to introduce a new module. Will the new module is
> available in the distribution package? I agree with Jark that we don't need
> to introduce this for table-API users and these users have their main
> class. If we want to make users write the k8s operator more easily, I think
> we should modify the k8s operator repo. If we don't need to support SQL
> files, can we make this jar only visible in the sql-gateway like we do in
> the planner loader?[1]
>
> [1]
>
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
>
> Best,
> Shengkai
>
>
>
>
>
>
>
>
> Weihua Hu <hu...@gmail.com> 于2023年6月7日周三 10:52写道：
>
> > Hi,
> >
> > Thanks for updating the FLIP.
> >
> > I have two cents on the distribution of SQLs and resources.
> > 1. Should we support a common file distribution mechanism for k8s
> > application mode?
> > I have seen some issues and requirements on the mailing list.
> > In our production environment, we implement the download command in the
> > CliFrontend.
> > And automatically add an init container to the POD for file
> downloading.
> > The advantage of this
> > is that we can use all Flink-supported file systems to store files.
> >
> > This need more discussion. I would appreciate hearing more opinions.
> >
> > 2. In this FLIP, we distribute files in two different ways in YARN and
> > Kubernetes. Can we combine it in one way?
> > If we don't want to implement a common file distribution for k8s
> > application mode. Could we use the SQLDriver
> > to download the files both in YARN and K8S? IMO, this can reduce the
> cost
> > of code maintenance.
> >
> > Best,
> > Weihua
> >
> >
> > On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
> >
> > > Hi Mason,
> > >
> > > Thanks for your input!
> > >
> > > > +1 for init containers or a more generalized way of obtaining
> arbitrary
> > > > files. File fetching isn't specific to just SQL--it also matters for
> > Java
> > > > applications if the user doesn't want to rebuild a Flink image and
> just
> > > > wants to modify the user application fat jar.
> > >
> > > I agree that utilizing SQL Drivers in Java applications is equally
> > > important
> > > as employing them in SQL Gateway. WRT init containers, I think most
> > > users use them just as a workaround. For example, wget a jar from the
> > > maven repo.
> > >
> > > We could implement the functionality in SQL Driver in a more graceful
> > > way and the flink-supported filesystem approach seems to be a
> > > good choice.
> > >
> > > > Also, what do you think about prefixing the config options with
> > > > `sql-driver` instead of just `sql` to be more specific?
> > >
> > > LGTM, since SQL Driver is a public interface and the options are
> > > specific to it.
> > >
> > > Best,
> > > Paul Lam
> > >
> > > > 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
> > > >
> > > > Hi Paul,
> > > >
> > > > +1 for this feature and supporting SQL file + JSON plans. We get a
> lot
> > of
> > > > requests to just be able to submit a SQL file, but the JSON plan
> > > > optimizations make sense.
> > > >
> > > > +1 for init containers or a more generalized way of obtaining
> arbitrary
> > > > files. File fetching isn't specific to just SQL--it also matters for
> > Java
> > > > applications if the user doesn't want to rebuild a Flink image and
> just
> > > > wants to modify the user application fat jar.
> > > >
> > > > Please note that we could reuse the checkpoint storage like S3/HDFS,
> > > which
> > > >> should
> > > >
> > > > be required to run Flink in production, so I guess that would be
> > > acceptable
> > > >> for most
> > > >
> > > > users. WDYT?
> > > >
> > > >
> > > > If you do go this route, it would be nice to support writing these
> > files
> > > to
> > > > S3/HDFS via Flink. This makes access control and policy management
> > > simpler.
> > > >
> > > > Also, what do you think about prefixing the config options with
> > > > `sql-driver` instead of just `sql` to be more specific?
> > > >
> > > > Best,
> > > > Mason
> > > >
> > > > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
> > <mailto:
> > > paullin3280@gmail.com>> wrote:
> > > >
> > > >> Hi Jark,
> > > >>
> > > >> Thanks for your input! Please see my comments inline.
> > > >>
> > > >>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > > >>> DataStream API also doesn't provide a default main class for users,
> > > >>> why do we need to provide such one for SQL?
> > > >>
> > > >> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> > > submitted
> > > >> via Flink CLI which actually could be DataStream/Table jobs.
> > > >>
> > > >> I think a default main class would be user-friendly which eliminates
> > the
> > > >> need
> > > >> for users to write a main class as SQLRunner in Flink K8s operator
> > [1].
> > > >>
> > > >>> I thought the proposed SqlDriver was a dedicated main class
> accepting
> > > >> SQL files, is
> > > >>> that correct?
> > > >>
> > > >> Both JSON plans and SQL files are accepted. SQL Gateway should use
> > JSON
> > > >> plans,
> > > >> while CLI users may use either JSON plans or SQL files.
> > > >>
> > > >> Please see the updated FLIP[2] for more details.
> > > >>
> > > >>> Personally, I prefer the way of init containers which doesn't
> depend
> > on
> > > >>> additional components.
> > > >>> This can reduce the moving parts of a production environment.
> > > >>> Depending on a distributed file system makes the testing, demo, and
> > > local
> > > >>> setup harder than init containers.
> > > >>
> > > >> Please note that we could reuse the checkpoint storage like S3/HDFS,
> > > which
> > > >> should
> > > >> be required to run Flink in production, so I guess that would be
> > > >> acceptable for most
> > > >> users. WDYT?
> > > >>
> > > >> WRT testing, demo, and local setups, I think we could support the
> > local
> > > >> filesystem
> > > >> scheme i.e. file://** as the state backends do. It works as long as
> > SQL
> > > >> Gateway
> > > >> and JobManager(or SQL Driver) can access the resource directory
> > > (specified
> > > >> via
> > > >> `sql-gateway.application.storage-dir`).
> > > >>
> > > >> Thanks!
> > > >>
> > > >> [1]
> > > >>
> > >
> >
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> > > >> [2]
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > > >> [3]
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> > > >>
> > > >> Best,
> > > >> Paul Lam
> > > >>
> > > >>> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
> > > >>>
> > > >>> Hi Paul,
> > > >>>
> > > >>> Thanks for your reply. I left my comments inline.
> > > >>>
> > > >>>> As the FLIP said, it’s good to have a default main class for Flink
> > > SQLs,
> > > >>>> which allows users to submit Flink SQLs in the same way as
> > DataStream
> > > >>>> jobs, or else users need to write their own main class.
> > > >>>
> > > >>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > > >>> DataStream API also doesn't provide a default main class for users,
> > > >>> why do we need to provide such one for SQL?
> > > >>>
> > > >>>> With the help of ExecNodeGraph, do we still need the serialized
> > > >>>> SessionState? If not, we could make SQL Driver accepts two
> > serialized
> > > >>>> formats:
> > > >>>
> > > >>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
> > the
> > > >>> proposed SqlDriver was a dedicated main class accepting SQL files,
> is
> > > >>> that correct?
> > > >>> If true, we have to ship the SessionState for this case which is a
> > > large
> > > >>> work.
> > > >>> I think we just need a JsonPlanDriver which is a main class that
> > > accepts
> > > >>> JsonPlan as the parameter.
> > > >>>
> > > >>>
> > > >>>> The common solutions I know is to use distributed file systems or
> > use
> > > >>>> init containers to localize the resources.
> > > >>>
> > > >>> Personally, I prefer the way of init containers which doesn't
> depend
> > on
> > > >>> additional components.
> > > >>> This can reduce the moving parts of a production environment.
> > > >>> Depending on a distributed file system makes the testing, demo, and
> > > local
> > > >>> setup harder than init containers.
> > > >>>
> > > >>> Best,
> > > >>> Jark
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
> > <mailto:
> > > paullin3280@gmail.com> <mailto:
> > > >> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
> > > >>>
> > > >>>> The FLIP is in the early phase and some details are not included,
> > but
> > > >>>> fortunately, we got lots of valuable ideas from the discussion.
> > > >>>>
> > > >>>> Thanks to everyone who joined the dissuasion!
> > > >>>> @Weihua @Shanmon @Shengkai @Biao @Jark
> > > >>>>
> > > >>>> This weekend I’m gonna revisit and update the FLIP, adding more
> > > >>>> details. Hopefully, we can further align our opinions.
> > > >>>>
> > > >>>> Best,
> > > >>>> Paul Lam
> > > >>>>
> > > >>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
> > > paullin3280@gmail.com>> 写道：
> > > >>>>>
> > > >>>>> Hi Jark,
> > > >>>>>
> > > >>>>> Thanks a lot for your input!
> > > >>>>>
> > > >>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> > > still
> > > >>>>>> necessary to support SQL Driver?
> > > >>>>>
> > > >>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
> > > simplify
> > > >>>>> Flink SQL execution with Flink CLI.
> > > >>>>>
> > > >>>>> As the FLIP said, it’s good to have a default main class for
> Flink
> > > >> SQLs,
> > > >>>>> which allows users to submit Flink SQLs in the same way as
> > DataStream
> > > >>>>> jobs, or else users need to write their own main class.
> > > >>>>>
> > > >>>>>> SQL Driver needs to serialize SessionState which is very
> > challenging
> > > >>>>>> but not detailed covered in the FLIP.
> > > >>>>>
> > > >>>>> With the help of ExecNodeGraph, do we still need the serialized
> > > >>>>> SessionState? If not, we could make SQL Driver accepts two
> > serialized
> > > >>>>> formats:
> > > >>>>>
> > > >>>>> - SQL files for user-facing public usage
> > > >>>>> - ExecNodeGraph for internal usage
> > > >>>>>
> > > >>>>> It’s kind of similar to the relationship between job jars and
> > > >> jobgraphs.
> > > >>>>>
> > > >>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> > > true?
> > > >>>> Is it
> > > >>>>>> possible to support it?
> > > >>>>>
> > > >>>>> Yes, K8s doesn’t distribute any files. It’s the users’
> > responsibility
> > > >> to
> > > >>>> make
> > > >>>>> sure the resources are accessible in the containers. The common
> > > >> solutions
> > > >>>>> I know is to use distributed file systems or use init containers
> to
> > > >>>> localize the
> > > >>>>> resources.
> > > >>>>>
> > > >>>>> Now I lean toward introducing a fs to do the distribution job.
> > WDYT?
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Paul Lam
> > > >>>>>
> > > >>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
> > imjark@gmail.com>
> > > <mailto:imjark@gmail.com <ma...@gmail.com>>
> > > >> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
> > > imjark@gmail.com <ma...@gmail.com>>>>
> > > >>>> 写道：
> > > >>>>>>
> > > >>>>>> Hi Paul,
> > > >>>>>>
> > > >>>>>> Thanks for starting this discussion. I like the proposal! This
> is
> > a
> > > >>>>>> frequently requested feature!
> > > >>>>>>
> > > >>>>>> I agree with Shengkai that ExecNodeGraph as the submission
> object
> > > is a
> > > >>>>>> better idea than SQL file. To be more specific, it should be
> > > >>>> JsonPlanGraph
> > > >>>>>> or CompiledPlan which is the serializable representation.
> > > CompiledPlan
> > > >>>> is a
> > > >>>>>> clear separation between compiling/optimization/validation and
> > > >>>> execution.
> > > >>>>>> This can keep the validation and metadata accessing still on the
> > > >>>> SQLGateway
> > > >>>>>> side. This allows SQLGateway to leverage some metadata caching
> and
> > > UDF
> > > >>>> JAR
> > > >>>>>> caching for better compiling performance.
> > > >>>>>>
> > > >>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> > > still
> > > >>>>>> necessary to support SQL Driver? Regarding non-interactive SQL
> > jobs,
> > > >>>> users
> > > >>>>>> can use the Table API program for application mode. SQL Driver
> > needs
> > > >> to
> > > >>>>>> serialize SessionState which is very challenging but not
> detailed
> > > >>>> covered
> > > >>>>>> in the FLIP.
> > > >>>>>>
> > > >>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> > > true?
> > > >>>> Is it
> > > >>>>>> possible to support it?
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Jark
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
> > > <ma...@gmail.com> <mailto:
> > > >> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> > > >>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:
> > > paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi Weihua,
> > > >>>>>>>
> > > >>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
> > > >>>> challenging
> > > >>>>>>> parts of this FLIP.
> > > >>>>>>>
> > > >>>>>>> Web submission is not enabled in application mode currently as
> > you
> > > >>>> said,
> > > >>>>>>> but it could be changed if we have good reasons.
> > > >>>>>>>
> > > >>>>>>> What do you think about introducing a distributed storage for
> SQL
> > > >>>> Gateway?
> > > >>>>>>>
> > > >>>>>>> We could make use of Flink file systems [1] to distribute the
> SQL
> > > >>>> Gateway
> > > >>>>>>> generated resources, that should solve the problem at its root
> > > cause.
> > > >>>>>>>
> > > >>>>>>> Users could specify Flink-supported file systems to ship files.
> > > It’s
> > > >>>> only
> > > >>>>>>> required when using SQL Gateway with K8s application mode.
> > > >>>>>>>
> > > >>>>>>> [1]
> > > >>>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > >
> > > >> <
> > > >>
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > >
> > > >>>
> > > >>>> <
> > > >>>>
> > > >>
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > >
> > > >> <
> > > >>
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > <
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > > >
> > > >>>
> > > >>>>>
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Paul Lam
> > > >>>>>>>
> > > >>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> > > huweihua.ckl@gmail.com> <mailto:
> > > >> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
> > > >>>>>>>>
> > > >>>>>>>> Thanks Paul for your reply.
> > > >>>>>>>>
> > > >>>>>>>> SQLDriver looks good to me.
> > > >>>>>>>>
> > > >>>>>>>> 2. Do you mean a pass the SQL string a configuration or a
> > program
> > > >>>>>>> argument?
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> I brought this up because we were unable to pass the SQL file
> to
> > > >> Flink
> > > >>>>>>>> using Kubernetes mode.
> > > >>>>>>>> For DataStream/Python users, they need to prepare their images
> > for
> > > >> the
> > > >>>>>>> jars
> > > >>>>>>>> and dependencies.
> > > >>>>>>>> But for SQL users, they can use a common image to run
> different
> > > SQL
> > > >>>>>>> queries
> > > >>>>>>>> if there are no other udf requirements.
> > > >>>>>>>> It would be great if the SQL query and image were not bound.
> > > >>>>>>>>
> > > >>>>>>>> Using strings is a way to decouple these, but just as you
> > > mentioned,
> > > >>>> it's
> > > >>>>>>>> not easy to pass complex SQL.
> > > >>>>>>>>
> > > >>>>>>>>> use web submission
> > > >>>>>>>> AFAIK, we can not use web submission in the Application mode.
> > > Please
> > > >>>>>>>> correct me if I'm wrong.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Best,
> > > >>>>>>>> Weihua
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <
> paullin3280@gmail.com
> > > <ma...@gmail.com>
> > > >> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> > > >>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi Biao,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for your comments!
> > > >>>>>>>>>
> > > >>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
> Flink
> > > SQL
> > > >>>> jobs
> > > >>>>>>>>> in
> > > >>>>>>>>>> Application mode? More specifically, if we use SQL
> > > client/gateway
> > > >> to
> > > >>>>>>>>>> execute some interactive SQLs like a SELECT query, can we
> ask
> > > >> flink
> > > >>>> to
> > > >>>>>>>>> use
> > > >>>>>>>>>> Application mode to execute those queries after this FLIP?
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for pointing it out. I think only DMLs would be
> executed
> > > via
> > > >>>> SQL
> > > >>>>>>>>> Driver.
> > > >>>>>>>>> I'll add the scope to the FLIP.
> > > >>>>>>>>>
> > > >>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> > > >>>> trivial as
> > > >>>>>>>>> we
> > > >>>>>>>>>> can ship files via YARN's tool easily but for K8s, things
> can
> > be
> > > >>>> more
> > > >>>>>>>>>> complicated as Shengkai said.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Your input is very informative. I’m thinking about using web
> > > >>>> submission,
> > > >>>>>>>>> but it requires exposing the JobManager port which could also
> > be
> > > a
> > > >>>>>>> problem
> > > >>>>>>>>> on K8s.
> > > >>>>>>>>>
> > > >>>>>>>>> Another approach is to explicitly require a distributed
> storage
> > > to
> > > >>>> ship
> > > >>>>>>>>> files,
> > > >>>>>>>>> but we may need a new deployment executor for that.
> > > >>>>>>>>>
> > > >>>>>>>>> What do you think of these two approaches?
> > > >>>>>>>>>
> > > >>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
> > > some
> > > >>>>>>>>>> unserializable fields
> > > >>>>>>>>>> like
> > > >>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> > > >>>>>>> It
> > > >>>>>>>>>> may be worthwhile to add more details about the
> serialization
> > > >> part.
> > > >>>>>>>>>
> > > >>>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph
> as
> > > >>>> Shengkai
> > > >>>>>>>>> mentioned, do we eliminate the need for serialization of
> > > >>>> SessionState?
> > > >>>>>>>>>
> > > >>>>>>>>> Best,
> > > >>>>>>>>> Paul Lam
> > > >>>>>>>>>
> > > >>>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
> > > biaogeng7@gmail.com> <mailto:
> > > >> biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks Paul for the proposal!I believe it would be very
> useful
> > > for
> > > >>>>>>> flink
> > > >>>>>>>>>> users.
> > > >>>>>>>>>> After reading the FLIP, I have some questions:
> > > >>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive
> Flink
> > > SQL
> > > >>>> jobs
> > > >>>>>>>>> in
> > > >>>>>>>>>> Application mode? More specifically, if we use SQL
> > > client/gateway
> > > >> to
> > > >>>>>>>>>> execute some interactive SQLs like a SELECT query, can we
> ask
> > > >> flink
> > > >>>> to
> > > >>>>>>>>> use
> > > >>>>>>>>>> Application mode to execute those queries after this FLIP?
> > > >>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> > > >>>> trivial as
> > > >>>>>>>>> we
> > > >>>>>>>>>> can ship files via YARN's tool easily but for K8s, things
> can
> > be
> > > >>>> more
> > > >>>>>>>>>> complicated as Shengkai said. I have implemented a simple
> POC
> > > >>>>>>>>>> <
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > > <
> > >
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > > >
> > > >> <
> > > >>
> > >
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > > <
> > >
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > > >
> > > >>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> based on SQL client before(i.e. consider the SQL client
> which
> > > >>>> supports
> > > >>>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One
> > > problem
> > > >> I
> > > >>>>>>> have
> > > >>>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s
> > side.
> > > >>>>>>> Without
> > > >>>>>>>>>> such support, users have to modify the initContainer or
> > rebuild
> > > a
> > > >>>> new
> > > >>>>>>> K8s
> > > >>>>>>>>>> image every time to fetch the SQL file. Like the flink k8s
> > > >> operator,
> > > >>>>>>> one
> > > >>>>>>>>>> workaround is to utilize the flink config(transforming the
> SQL
> > > >> file
> > > >>>> to
> > > >>>>>>> a
> > > >>>>>>>>>> escaped string like Weihua mentioned) which will be
> converted
> > > to a
> > > >>>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger
> than
> > > 1MB
> > > >>>>>>>>>> <
> https://kubernetes.io/docs/concepts/configuration/configmap/
> > <
> > > https://kubernetes.io/docs/concepts/configuration/configmap/> <
> > > >> https://kubernetes.io/docs/concepts/configuration/configmap/ <
> > > https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
> > > >>>> Not
> > > >>>>>>>>> sure
> > > >>>>>>>>>> if we have better solutions.
> > > >>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
> > > some
> > > >>>>>>>>>> unserializable fields
> > > >>>>>>>>>> like
> > > >>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> > > >>>>>>> It
> > > >>>>>>>>>> may be worthwhile to add more details about the
> serialization
> > > >> part.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best,
> > > >>>>>>>>>> Biao Geng
> > > >>>>>>>>>>
> > > >>>>>>>>>> Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com
> > >
> > > <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> > > >> 于2023年5月31日周三 11:49写道：
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi Weihua,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks a lot for your input! Please see my comments inline.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
> > Job.
> > > >> (Not
> > > >>>>>>>>>>> strong,
> > > >>>>>>>>>>>> the SQLDriver is fine for me)
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
> > > >>>> following
> > > >>>>>>>>>>> reasons FYI:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
> > > >>>>>>>>>>> 2. Flink program's main class is sort of like Driver in
> JDBC
> > > >> which
> > > >>>>>>>>>>> translates SQLs into
> > > >>>>>>>>>>> databases specific languages.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
> we
> > > need
> > > >>>> to
> > > >>>>>>>>>>> prepare
> > > >>>>>>>>>>>> a SQL file in an image for Kubernetes application mode,
> > which
> > > >> may
> > > >>>> be
> > > >>>>>>> a
> > > >>>>>>>>>>> bit
> > > >>>>>>>>>>>> cumbersome.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Do you mean a pass the SQL string a configuration or a
> > program
> > > >>>>>>> argument?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I thought it might be convenient for testing propose, but
> not
> > > >>>>>>>>> recommended
> > > >>>>>>>>>>> for production,
> > > >>>>>>>>>>> cause Flink SQLs could be complicated and involves lots of
> > > >>>> characters
> > > >>>>>>>>> that
> > > >>>>>>>>>>> need to escape.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> > > >>>>>>>>>>> "run-application"
> > > >>>>>>>>>>>> command. Does that mean we need to perform automatic
> > detection
> > > >> in
> > > >>>>>>>>> Flink?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Yes! It’s like running a PyFlink job with the following
> > > command:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> ```
> > > >>>>>>>>>>> ./bin/flink run \
> > > >>>>>>>>>>> --pyModule table.word_count \
> > > >>>>>>>>>>> --pyFiles examples/python/table
> > > >>>>>>>>>>> ```
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
> > > Driver
> > > >>>>>>>>>>> automatically.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [1]
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > > <
> > >
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > > >
> > > >> <
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > > <
> > >
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > > >
> > > >>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best,
> > > >>>>>>>>>>> Paul Lam
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com
> <mailto:
> > > huweihua.ckl@gmail.com> <mailto:
> > > >> huweihua.ckl@gmail.com>> 写道：
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks Paul for the proposal.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> +1 for this. It is valuable in improving ease of use.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I have a few questions.
> > > >>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
> > Job.
> > > >> (Not
> > > >>>>>>>>>>> strong,
> > > >>>>>>>>>>>> the SQLDriver is fine for me)
> > > >>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise,
> we
> > > need
> > > >>>> to
> > > >>>>>>>>>>> prepare
> > > >>>>>>>>>>>> a SQL file in an image for Kubernetes application mode,
> > which
> > > >> may
> > > >>>> be
> > > >>>>>>> a
> > > >>>>>>>>>>> bit
> > > >>>>>>>>>>>> cumbersome.
> > > >>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> > > >>>>>>>>>>> "run-application"
> > > >>>>>>>>>>>> command. Does that mean we need to perform automatic
> > detection
> > > >> in
> > > >>>>>>>>> Flink?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> Weihua
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> > > paullin3280@gmail.com
> > > >> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi team,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
> > > >>>> introduces
> > > >>>>>>> a
> > > >>>>>>>>>>> SQL
> > > >>>>>>>>>>>>> driver as the
> > > >>>>>>>>>>>>> default main class for Flink SQL jobs.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Currently, Flink SQL could be executed out of the box
> > either
> > > >> via
> > > >>>> SQL
> > > >>>>>>>>>>>>> Client/Gateway
> > > >>>>>>>>>>>>> or embedded in a Flink Java/Python program.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> However, each one has its drawback:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application
> > > deployment
> > > >>>> mode
> > > >>>>>>>>> [2]
> > > >>>>>>>>>>>>> - Flink Java/Python program requires extra work to write
> a
> > > >>>> non-SQL
> > > >>>>>>>>>>> program
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the
> > > default
> > > >>>> main
> > > >>>>>>>>>>> class
> > > >>>>>>>>>>>>> for SQL jobs.
> > > >>>>>>>>>>>>> Please see the FLIP docs for details and feel free to
> > > comment.
> > > >>>>>>> Thanks!
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> > > >>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>> Paul Lam
> > >
> > >
> >
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Shengkai Fang <fs...@gmail.com>.
Hi. Paul. Thanks for your update and the update makes me understand the
design much better.
But I still have some questions about the FLIP.
> For SQL Gateway, only DMLs need to be delegated to the SQL server
> Driver. I would think about the details and update the FLIP. Do you have
some
> ideas already?
If the applicaiton mode can not support library mode, I think we should
only execute INSERT INTO and UPDATE/ DELETE statement in the application
mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE
statements. The ANALYZE TABLE syntax need to register the statistic to the
catalog after job finishes and the CALL PROCEDURE statement doesn't
generate the ExecNodeGraph.
* Introduce storage via option `sql-gateway.application.storage-dir`
If we can not support to submit the jars through web submission, +1 to
introduce the options to upload the files. While I think the uploader
should be responsible to remove the uploaded jars. Can we remove the jars
if the job is running or gateway exits?
* JobID is not avaliable
Can we use the returned rest client by ApplicationDeployer to query the job
id? I am concerned that users don't know which job is related to the
submitted SQL.
* Do we need to introduce a new module named flink-table-sql-runner?
It seems we need to introduce a new module. Will the new module is
available in the distribution package? I agree with Jark that we don't need
to introduce this for table-API users and these users have their main
class. If we want to make users write the k8s operator more easily, I think
we should modify the k8s operator repo. If we don't need to support SQL
files, can we make this jar only visible in the sql-gateway like we do in
the planner loader?[1]
[1]
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95
Best,
Shengkai
Weihua Hu <hu...@gmail.com> 于2023年6月7日周三 10:52写道：
> Hi,
>
> Thanks for updating the FLIP.
>
> I have two cents on the distribution of SQLs and resources.
> 1. Should we support a common file distribution mechanism for k8s
> application mode?
> I have seen some issues and requirements on the mailing list.
> In our production environment, we implement the download command in the
> CliFrontend.
> And automatically add an init container to the POD for file downloading.
> The advantage of this
> is that we can use all Flink-supported file systems to store files.
>
> This need more discussion. I would appreciate hearing more opinions.
>
> 2. In this FLIP, we distribute files in two different ways in YARN and
> Kubernetes. Can we combine it in one way?
> If we don't want to implement a common file distribution for k8s
> application mode. Could we use the SQLDriver
> to download the files both in YARN and K8S? IMO, this can reduce the cost
> of code maintenance.
>
> Best,
> Weihua
>
>
> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
>
> > Hi Mason,
> >
> > Thanks for your input!
> >
> > > +1 for init containers or a more generalized way of obtaining arbitrary
> > > files. File fetching isn't specific to just SQL--it also matters for
> Java
> > > applications if the user doesn't want to rebuild a Flink image and just
> > > wants to modify the user application fat jar.
> >
> > I agree that utilizing SQL Drivers in Java applications is equally
> > important
> > as employing them in SQL Gateway. WRT init containers, I think most
> > users use them just as a workaround. For example, wget a jar from the
> > maven repo.
> >
> > We could implement the functionality in SQL Driver in a more graceful
> > way and the flink-supported filesystem approach seems to be a
> > good choice.
> >
> > > Also, what do you think about prefixing the config options with
> > > `sql-driver` instead of just `sql` to be more specific?
> >
> > LGTM, since SQL Driver is a public interface and the options are
> > specific to it.
> >
> > Best,
> > Paul Lam
> >
> > > 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
> > >
> > > Hi Paul,
> > >
> > > +1 for this feature and supporting SQL file + JSON plans. We get a lot
> of
> > > requests to just be able to submit a SQL file, but the JSON plan
> > > optimizations make sense.
> > >
> > > +1 for init containers or a more generalized way of obtaining arbitrary
> > > files. File fetching isn't specific to just SQL--it also matters for
> Java
> > > applications if the user doesn't want to rebuild a Flink image and just
> > > wants to modify the user application fat jar.
> > >
> > > Please note that we could reuse the checkpoint storage like S3/HDFS,
> > which
> > >> should
> > >
> > > be required to run Flink in production, so I guess that would be
> > acceptable
> > >> for most
> > >
> > > users. WDYT?
> > >
> > >
> > > If you do go this route, it would be nice to support writing these
> files
> > to
> > > S3/HDFS via Flink. This makes access control and policy management
> > simpler.
> > >
> > > Also, what do you think about prefixing the config options with
> > > `sql-driver` instead of just `sql` to be more specific?
> > >
> > > Best,
> > > Mason
> > >
> > > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com
> <mailto:
> > paullin3280@gmail.com>> wrote:
> > >
> > >> Hi Jark,
> > >>
> > >> Thanks for your input! Please see my comments inline.
> > >>
> > >>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > >>> DataStream API also doesn't provide a default main class for users,
> > >>> why do we need to provide such one for SQL?
> > >>
> > >> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> > submitted
> > >> via Flink CLI which actually could be DataStream/Table jobs.
> > >>
> > >> I think a default main class would be user-friendly which eliminates
> the
> > >> need
> > >> for users to write a main class as SQLRunner in Flink K8s operator
> [1].
> > >>
> > >>> I thought the proposed SqlDriver was a dedicated main class accepting
> > >> SQL files, is
> > >>> that correct?
> > >>
> > >> Both JSON plans and SQL files are accepted. SQL Gateway should use
> JSON
> > >> plans,
> > >> while CLI users may use either JSON plans or SQL files.
> > >>
> > >> Please see the updated FLIP[2] for more details.
> > >>
> > >>> Personally, I prefer the way of init containers which doesn't depend
> on
> > >>> additional components.
> > >>> This can reduce the moving parts of a production environment.
> > >>> Depending on a distributed file system makes the testing, demo, and
> > local
> > >>> setup harder than init containers.
> > >>
> > >> Please note that we could reuse the checkpoint storage like S3/HDFS,
> > which
> > >> should
> > >> be required to run Flink in production, so I guess that would be
> > >> acceptable for most
> > >> users. WDYT?
> > >>
> > >> WRT testing, demo, and local setups, I think we could support the
> local
> > >> filesystem
> > >> scheme i.e. file://** as the state backends do. It works as long as
> SQL
> > >> Gateway
> > >> and JobManager(or SQL Driver) can access the resource directory
> > (specified
> > >> via
> > >> `sql-gateway.application.storage-dir`).
> > >>
> > >> Thanks!
> > >>
> > >> [1]
> > >>
> >
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> > >> [2]
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > >> [3]
> > >>
> >
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> > >>
> > >> Best,
> > >> Paul Lam
> > >>
> > >>> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
> > >>>
> > >>> Hi Paul,
> > >>>
> > >>> Thanks for your reply. I left my comments inline.
> > >>>
> > >>>> As the FLIP said, it’s good to have a default main class for Flink
> > SQLs,
> > >>>> which allows users to submit Flink SQLs in the same way as
> DataStream
> > >>>> jobs, or else users need to write their own main class.
> > >>>
> > >>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > >>> DataStream API also doesn't provide a default main class for users,
> > >>> why do we need to provide such one for SQL?
> > >>>
> > >>>> With the help of ExecNodeGraph, do we still need the serialized
> > >>>> SessionState? If not, we could make SQL Driver accepts two
> serialized
> > >>>> formats:
> > >>>
> > >>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought
> the
> > >>> proposed SqlDriver was a dedicated main class accepting SQL files, is
> > >>> that correct?
> > >>> If true, we have to ship the SessionState for this case which is a
> > large
> > >>> work.
> > >>> I think we just need a JsonPlanDriver which is a main class that
> > accepts
> > >>> JsonPlan as the parameter.
> > >>>
> > >>>
> > >>>> The common solutions I know is to use distributed file systems or
> use
> > >>>> init containers to localize the resources.
> > >>>
> > >>> Personally, I prefer the way of init containers which doesn't depend
> on
> > >>> additional components.
> > >>> This can reduce the moving parts of a production environment.
> > >>> Depending on a distributed file system makes the testing, demo, and
> > local
> > >>> setup harder than init containers.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com
> <mailto:
> > paullin3280@gmail.com> <mailto:
> > >> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
> > >>>
> > >>>> The FLIP is in the early phase and some details are not included,
> but
> > >>>> fortunately, we got lots of valuable ideas from the discussion.
> > >>>>
> > >>>> Thanks to everyone who joined the dissuasion!
> > >>>> @Weihua @Shanmon @Shengkai @Biao @Jark
> > >>>>
> > >>>> This weekend I’m gonna revisit and update the FLIP, adding more
> > >>>> details. Hopefully, we can further align our opinions.
> > >>>>
> > >>>> Best,
> > >>>> Paul Lam
> > >>>>
> > >>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
> > paullin3280@gmail.com>> 写道：
> > >>>>>
> > >>>>> Hi Jark,
> > >>>>>
> > >>>>> Thanks a lot for your input!
> > >>>>>
> > >>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> > still
> > >>>>>> necessary to support SQL Driver?
> > >>>>>
> > >>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
> > simplify
> > >>>>> Flink SQL execution with Flink CLI.
> > >>>>>
> > >>>>> As the FLIP said, it’s good to have a default main class for Flink
> > >> SQLs,
> > >>>>> which allows users to submit Flink SQLs in the same way as
> DataStream
> > >>>>> jobs, or else users need to write their own main class.
> > >>>>>
> > >>>>>> SQL Driver needs to serialize SessionState which is very
> challenging
> > >>>>>> but not detailed covered in the FLIP.
> > >>>>>
> > >>>>> With the help of ExecNodeGraph, do we still need the serialized
> > >>>>> SessionState? If not, we could make SQL Driver accepts two
> serialized
> > >>>>> formats:
> > >>>>>
> > >>>>> - SQL files for user-facing public usage
> > >>>>> - ExecNodeGraph for internal usage
> > >>>>>
> > >>>>> It’s kind of similar to the relationship between job jars and
> > >> jobgraphs.
> > >>>>>
> > >>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> > true?
> > >>>> Is it
> > >>>>>> possible to support it?
> > >>>>>
> > >>>>> Yes, K8s doesn’t distribute any files. It’s the users’
> responsibility
> > >> to
> > >>>> make
> > >>>>> sure the resources are accessible in the containers. The common
> > >> solutions
> > >>>>> I know is to use distributed file systems or use init containers to
> > >>>> localize the
> > >>>>> resources.
> > >>>>>
> > >>>>> Now I lean toward introducing a fs to do the distribution job.
> WDYT?
> > >>>>>
> > >>>>> Best,
> > >>>>> Paul Lam
> > >>>>>
> > >>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <mailto:
> imjark@gmail.com>
> > <mailto:imjark@gmail.com <ma...@gmail.com>>
> > >> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
> > imjark@gmail.com <ma...@gmail.com>>>>
> > >>>> 写道：
> > >>>>>>
> > >>>>>> Hi Paul,
> > >>>>>>
> > >>>>>> Thanks for starting this discussion. I like the proposal! This is
> a
> > >>>>>> frequently requested feature!
> > >>>>>>
> > >>>>>> I agree with Shengkai that ExecNodeGraph as the submission object
> > is a
> > >>>>>> better idea than SQL file. To be more specific, it should be
> > >>>> JsonPlanGraph
> > >>>>>> or CompiledPlan which is the serializable representation.
> > CompiledPlan
> > >>>> is a
> > >>>>>> clear separation between compiling/optimization/validation and
> > >>>> execution.
> > >>>>>> This can keep the validation and metadata accessing still on the
> > >>>> SQLGateway
> > >>>>>> side. This allows SQLGateway to leverage some metadata caching and
> > UDF
> > >>>> JAR
> > >>>>>> caching for better compiling performance.
> > >>>>>>
> > >>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> > still
> > >>>>>> necessary to support SQL Driver? Regarding non-interactive SQL
> jobs,
> > >>>> users
> > >>>>>> can use the Table API program for application mode. SQL Driver
> needs
> > >> to
> > >>>>>> serialize SessionState which is very challenging but not detailed
> > >>>> covered
> > >>>>>> in the FLIP.
> > >>>>>>
> > >>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> > true?
> > >>>> Is it
> > >>>>>> possible to support it?
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Jark
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
> > <ma...@gmail.com> <mailto:
> > >> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> > >>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:
> > paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi Weihua,
> > >>>>>>>
> > >>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
> > >>>> challenging
> > >>>>>>> parts of this FLIP.
> > >>>>>>>
> > >>>>>>> Web submission is not enabled in application mode currently as
> you
> > >>>> said,
> > >>>>>>> but it could be changed if we have good reasons.
> > >>>>>>>
> > >>>>>>> What do you think about introducing a distributed storage for SQL
> > >>>> Gateway?
> > >>>>>>>
> > >>>>>>> We could make use of Flink file systems [1] to distribute the SQL
> > >>>> Gateway
> > >>>>>>> generated resources, that should solve the problem at its root
> > cause.
> > >>>>>>>
> > >>>>>>> Users could specify Flink-supported file systems to ship files.
> > It’s
> > >>>> only
> > >>>>>>> required when using SQL Gateway with K8s application mode.
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>>>>
> > >>>>
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > >> <
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > >>>
> > >>>> <
> > >>>>
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > >> <
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> > >
> > >>>
> > >>>>>
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Paul Lam
> > >>>>>>>
> > >>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> > huweihua.ckl@gmail.com> <mailto:
> > >> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
> > >>>>>>>>
> > >>>>>>>> Thanks Paul for your reply.
> > >>>>>>>>
> > >>>>>>>> SQLDriver looks good to me.
> > >>>>>>>>
> > >>>>>>>> 2. Do you mean a pass the SQL string a configuration or a
> program
> > >>>>>>> argument?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> I brought this up because we were unable to pass the SQL file to
> > >> Flink
> > >>>>>>>> using Kubernetes mode.
> > >>>>>>>> For DataStream/Python users, they need to prepare their images
> for
> > >> the
> > >>>>>>> jars
> > >>>>>>>> and dependencies.
> > >>>>>>>> But for SQL users, they can use a common image to run different
> > SQL
> > >>>>>>> queries
> > >>>>>>>> if there are no other udf requirements.
> > >>>>>>>> It would be great if the SQL query and image were not bound.
> > >>>>>>>>
> > >>>>>>>> Using strings is a way to decouple these, but just as you
> > mentioned,
> > >>>> it's
> > >>>>>>>> not easy to pass complex SQL.
> > >>>>>>>>
> > >>>>>>>>> use web submission
> > >>>>>>>> AFAIK, we can not use web submission in the Application mode.
> > Please
> > >>>>>>>> correct me if I'm wrong.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> Weihua
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <paullin3280@gmail.com
> > <ma...@gmail.com>
> > >> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> > >>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Biao,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for your comments!
> > >>>>>>>>>
> > >>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink
> > SQL
> > >>>> jobs
> > >>>>>>>>> in
> > >>>>>>>>>> Application mode? More specifically, if we use SQL
> > client/gateway
> > >> to
> > >>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
> > >> flink
> > >>>> to
> > >>>>>>>>> use
> > >>>>>>>>>> Application mode to execute those queries after this FLIP?
> > >>>>>>>>>
> > >>>>>>>>> Thanks for pointing it out. I think only DMLs would be executed
> > via
> > >>>> SQL
> > >>>>>>>>> Driver.
> > >>>>>>>>> I'll add the scope to the FLIP.
> > >>>>>>>>>
> > >>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> > >>>> trivial as
> > >>>>>>>>> we
> > >>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can
> be
> > >>>> more
> > >>>>>>>>>> complicated as Shengkai said.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Your input is very informative. I’m thinking about using web
> > >>>> submission,
> > >>>>>>>>> but it requires exposing the JobManager port which could also
> be
> > a
> > >>>>>>> problem
> > >>>>>>>>> on K8s.
> > >>>>>>>>>
> > >>>>>>>>> Another approach is to explicitly require a distributed storage
> > to
> > >>>> ship
> > >>>>>>>>> files,
> > >>>>>>>>> but we may need a new deployment executor for that.
> > >>>>>>>>>
> > >>>>>>>>> What do you think of these two approaches?
> > >>>>>>>>>
> > >>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
> > some
> > >>>>>>>>>> unserializable fields
> > >>>>>>>>>> like
> > >>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> > >>>>>>> It
> > >>>>>>>>>> may be worthwhile to add more details about the serialization
> > >> part.
> > >>>>>>>>>
> > >>>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as
> > >>>> Shengkai
> > >>>>>>>>> mentioned, do we eliminate the need for serialization of
> > >>>> SessionState?
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Paul Lam
> > >>>>>>>>>
> > >>>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
> > biaogeng7@gmail.com> <mailto:
> > >> biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks Paul for the proposal!I believe it would be very useful
> > for
> > >>>>>>> flink
> > >>>>>>>>>> users.
> > >>>>>>>>>> After reading the FLIP, I have some questions:
> > >>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink
> > SQL
> > >>>> jobs
> > >>>>>>>>> in
> > >>>>>>>>>> Application mode? More specifically, if we use SQL
> > client/gateway
> > >> to
> > >>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
> > >> flink
> > >>>> to
> > >>>>>>>>> use
> > >>>>>>>>>> Application mode to execute those queries after this FLIP?
> > >>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> > >>>> trivial as
> > >>>>>>>>> we
> > >>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can
> be
> > >>>> more
> > >>>>>>>>>> complicated as Shengkai said. I have implemented a simple POC
> > >>>>>>>>>> <
> > >>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > <
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > >
> > >> <
> > >>
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > <
> >
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> > >
> > >>>
> > >>>>>>>>>>
> > >>>>>>>>>> based on SQL client before(i.e. consider the SQL client which
> > >>>> supports
> > >>>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One
> > problem
> > >> I
> > >>>>>>> have
> > >>>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s
> side.
> > >>>>>>> Without
> > >>>>>>>>>> such support, users have to modify the initContainer or
> rebuild
> > a
> > >>>> new
> > >>>>>>> K8s
> > >>>>>>>>>> image every time to fetch the SQL file. Like the flink k8s
> > >> operator,
> > >>>>>>> one
> > >>>>>>>>>> workaround is to utilize the flink config(transforming the SQL
> > >> file
> > >>>> to
> > >>>>>>> a
> > >>>>>>>>>> escaped string like Weihua mentioned) which will be converted
> > to a
> > >>>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than
> > 1MB
> > >>>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/
> <
> > https://kubernetes.io/docs/concepts/configuration/configmap/> <
> > >> https://kubernetes.io/docs/concepts/configuration/configmap/ <
> > https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
> > >>>> Not
> > >>>>>>>>> sure
> > >>>>>>>>>> if we have better solutions.
> > >>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
> > some
> > >>>>>>>>>> unserializable fields
> > >>>>>>>>>> like
> > >>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> > >>>>>>> It
> > >>>>>>>>>> may be worthwhile to add more details about the serialization
> > >> part.
> > >>>>>>>>>>
> > >>>>>>>>>> Best,
> > >>>>>>>>>> Biao Geng
> > >>>>>>>>>>
> > >>>>>>>>>> Paul Lam <paullin3280@gmail.com <mailto:paullin3280@gmail.com
> >
> > <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> > >> 于2023年5月31日周三 11:49写道：
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Weihua,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks a lot for your input! Please see my comments inline.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
> Job.
> > >> (Not
> > >>>>>>>>>>> strong,
> > >>>>>>>>>>>> the SQLDriver is fine for me)
> > >>>>>>>>>>>
> > >>>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
> > >>>> following
> > >>>>>>>>>>> reasons FYI:
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
> > >>>>>>>>>>> 2. Flink program's main class is sort of like Driver in JDBC
> > >> which
> > >>>>>>>>>>> translates SQLs into
> > >>>>>>>>>>> databases specific languages.
> > >>>>>>>>>>>
> > >>>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we
> > need
> > >>>> to
> > >>>>>>>>>>> prepare
> > >>>>>>>>>>>> a SQL file in an image for Kubernetes application mode,
> which
> > >> may
> > >>>> be
> > >>>>>>> a
> > >>>>>>>>>>> bit
> > >>>>>>>>>>>> cumbersome.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Do you mean a pass the SQL string a configuration or a
> program
> > >>>>>>> argument?
> > >>>>>>>>>>>
> > >>>>>>>>>>> I thought it might be convenient for testing propose, but not
> > >>>>>>>>> recommended
> > >>>>>>>>>>> for production,
> > >>>>>>>>>>> cause Flink SQLs could be complicated and involves lots of
> > >>>> characters
> > >>>>>>>>> that
> > >>>>>>>>>>> need to escape.
> > >>>>>>>>>>>
> > >>>>>>>>>>> WDYT?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> > >>>>>>>>>>> "run-application"
> > >>>>>>>>>>>> command. Does that mean we need to perform automatic
> detection
> > >> in
> > >>>>>>>>> Flink?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Yes! It’s like running a PyFlink job with the following
> > command:
> > >>>>>>>>>>>
> > >>>>>>>>>>> ```
> > >>>>>>>>>>> ./bin/flink run \
> > >>>>>>>>>>> --pyModule table.word_count \
> > >>>>>>>>>>> --pyFiles examples/python/table
> > >>>>>>>>>>> ```
> > >>>>>>>>>>>
> > >>>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
> > Driver
> > >>>>>>>>>>> automatically.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> [1]
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > <
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > >
> > >> <
> > >>
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > <
> >
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> > >
> > >>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> Paul Lam
> > >>>>>>>>>>>
> > >>>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> > huweihua.ckl@gmail.com> <mailto:
> > >> huweihua.ckl@gmail.com>> 写道：
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks Paul for the proposal.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +1 for this. It is valuable in improving ease of use.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I have a few questions.
> > >>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL
> Job.
> > >> (Not
> > >>>>>>>>>>> strong,
> > >>>>>>>>>>>> the SQLDriver is fine for me)
> > >>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we
> > need
> > >>>> to
> > >>>>>>>>>>> prepare
> > >>>>>>>>>>>> a SQL file in an image for Kubernetes application mode,
> which
> > >> may
> > >>>> be
> > >>>>>>> a
> > >>>>>>>>>>> bit
> > >>>>>>>>>>>> cumbersome.
> > >>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> > >>>>>>>>>>> "run-application"
> > >>>>>>>>>>>> command. Does that mean we need to perform automatic
> detection
> > >> in
> > >>>>>>>>> Flink?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Weihua
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> > paullin3280@gmail.com
> > >> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi team,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
> > >>>> introduces
> > >>>>>>> a
> > >>>>>>>>>>> SQL
> > >>>>>>>>>>>>> driver as the
> > >>>>>>>>>>>>> default main class for Flink SQL jobs.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Currently, Flink SQL could be executed out of the box
> either
> > >> via
> > >>>> SQL
> > >>>>>>>>>>>>> Client/Gateway
> > >>>>>>>>>>>>> or embedded in a Flink Java/Python program.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> However, each one has its drawback:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application
> > deployment
> > >>>> mode
> > >>>>>>>>> [2]
> > >>>>>>>>>>>>> - Flink Java/Python program requires extra work to write a
> > >>>> non-SQL
> > >>>>>>>>>>> program
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the
> > default
> > >>>> main
> > >>>>>>>>>>> class
> > >>>>>>>>>>>>> for SQL jobs.
> > >>>>>>>>>>>>> Please see the FLIP docs for details and feel free to
> > comment.
> > >>>>>>> Thanks!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> > >>>>>>>>>>>>> <
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>> Paul Lam
> >
> >
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Weihua,
Thanks a lot for your input!
I see the difference here is implementing the file distribution mechanism
in the generic CLI or in the SQL Driver. The CLI approach could benefit
non-pure-SQL applications (which is not covered by SQL Driver) as well.
Not sure if you’re proposing the CliFrontend + init containers approach
or suggesting a native file mechanism for the flink-kubernetes module?
Considering your 2nd question, I’m assuming the latter. I think it’s totally
worthwhile to discuss, and we may split it into another FLIP, because
it’s a significant change.
Personally, I’m big +1 for a built-in file distribution mechanism on K8s.
Best,
Paul Lam
> 2023年6月7日 10:51，Weihua Hu <hu...@gmail.com> 写道：
>
> Hi,
>
> Thanks for updating the FLIP.
>
> I have two cents on the distribution of SQLs and resources.
> 1. Should we support a common file distribution mechanism for k8s
> application mode?
> I have seen some issues and requirements on the mailing list.
> In our production environment, we implement the download command in the
> CliFrontend.
> And automatically add an init container to the POD for file downloading.
> The advantage of this
> is that we can use all Flink-supported file systems to store files.
>
> This need more discussion. I would appreciate hearing more opinions.
>
> 2. In this FLIP, we distribute files in two different ways in YARN and
> Kubernetes. Can we combine it in one way?
> If we don't want to implement a common file distribution for k8s
> application mode. Could we use the SQLDriver
> to download the files both in YARN and K8S? IMO, this can reduce the cost
> of code maintenance.
>
> Best,
> Weihua
>
>
> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
>
>> Hi Mason,
>>
>> Thanks for your input!
>>
>>> +1 for init containers or a more generalized way of obtaining arbitrary
>>> files. File fetching isn't specific to just SQL--it also matters for Java
>>> applications if the user doesn't want to rebuild a Flink image and just
>>> wants to modify the user application fat jar.
>>
>> I agree that utilizing SQL Drivers in Java applications is equally
>> important
>> as employing them in SQL Gateway. WRT init containers, I think most
>> users use them just as a workaround. For example, wget a jar from the
>> maven repo.
>>
>> We could implement the functionality in SQL Driver in a more graceful
>> way and the flink-supported filesystem approach seems to be a
>> good choice.
>>
>>> Also, what do you think about prefixing the config options with
>>> `sql-driver` instead of just `sql` to be more specific?
>>
>> LGTM, since SQL Driver is a public interface and the options are
>> specific to it.
>>
>> Best,
>> Paul Lam
>>
>>> 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
>>>
>>> Hi Paul,
>>>
>>> +1 for this feature and supporting SQL file + JSON plans. We get a lot of
>>> requests to just be able to submit a SQL file, but the JSON plan
>>> optimizations make sense.
>>>
>>> +1 for init containers or a more generalized way of obtaining arbitrary
>>> files. File fetching isn't specific to just SQL--it also matters for Java
>>> applications if the user doesn't want to rebuild a Flink image and just
>>> wants to modify the user application fat jar.
>>>
>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>> which
>>>> should
>>>
>>> be required to run Flink in production, so I guess that would be
>> acceptable
>>>> for most
>>>
>>> users. WDYT?
>>>
>>>
>>> If you do go this route, it would be nice to support writing these files
>> to
>>> S3/HDFS via Flink. This makes access control and policy management
>> simpler.
>>>
>>> Also, what do you think about prefixing the config options with
>>> `sql-driver` instead of just `sql` to be more specific?
>>>
>>> Best,
>>> Mason
>>>
>>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>>>
>>>> Hi Jark,
>>>>
>>>> Thanks for your input! Please see my comments inline.
>>>>
>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>> DataStream API also doesn't provide a default main class for users,
>>>>> why do we need to provide such one for SQL?
>>>>
>>>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
>> submitted
>>>> via Flink CLI which actually could be DataStream/Table jobs.
>>>>
>>>> I think a default main class would be user-friendly which eliminates the
>>>> need
>>>> for users to write a main class as SQLRunner in Flink K8s operator [1].
>>>>
>>>>> I thought the proposed SqlDriver was a dedicated main class accepting
>>>> SQL files, is
>>>>> that correct?
>>>>
>>>> Both JSON plans and SQL files are accepted. SQL Gateway should use JSON
>>>> plans,
>>>> while CLI users may use either JSON plans or SQL files.
>>>>
>>>> Please see the updated FLIP[2] for more details.
>>>>
>>>>> Personally, I prefer the way of init containers which doesn't depend on
>>>>> additional components.
>>>>> This can reduce the moving parts of a production environment.
>>>>> Depending on a distributed file system makes the testing, demo, and
>> local
>>>>> setup harder than init containers.
>>>>
>>>> Please note that we could reuse the checkpoint storage like S3/HDFS,
>> which
>>>> should
>>>> be required to run Flink in production, so I guess that would be
>>>> acceptable for most
>>>> users. WDYT?
>>>>
>>>> WRT testing, demo, and local setups, I think we could support the local
>>>> filesystem
>>>> scheme i.e. file://** <file:///**> as the state backends do. It works as long as SQL
>>>> Gateway
>>>> and JobManager(or SQL Driver) can access the resource directory
>> (specified
>>>> via
>>>> `sql-gateway.application.storage-dir`).
>>>>
>>>> Thanks!
>>>>
>>>> [1]
>>>>
>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java <https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java>
>>>> [2]
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver <https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver>
>>>> [3]
>>>>
>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 <https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161>
>>>>
>>>> Best,
>>>> Paul Lam
>>>>
>>>>> 2023年6月3日 12:21，Jark Wu <imjark@gmail.com <ma...@gmail.com>> 写道：
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Thanks for your reply. I left my comments inline.
>>>>>
>>>>>> As the FLIP said, it’s good to have a default main class for Flink
>> SQLs,
>>>>>> which allows users to submit Flink SQLs in the same way as DataStream
>>>>>> jobs, or else users need to write their own main class.
>>>>>
>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>>>> DataStream API also doesn't provide a default main class for users,
>>>>> why do we need to provide such one for SQL?
>>>>>
>>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>>> SessionState? If not, we could make SQL Driver accepts two serialized
>>>>>> formats:
>>>>>
>>>>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought the
>>>>> proposed SqlDriver was a dedicated main class accepting SQL files, is
>>>>> that correct?
>>>>> If true, we have to ship the SessionState for this case which is a
>> large
>>>>> work.
>>>>> I think we just need a JsonPlanDriver which is a main class that
>> accepts
>>>>> JsonPlan as the parameter.
>>>>>
>>>>>
>>>>>> The common solutions I know is to use distributed file systems or use
>>>>>> init containers to localize the resources.
>>>>>
>>>>> Personally, I prefer the way of init containers which doesn't depend on
>>>>> additional components.
>>>>> This can reduce the moving parts of a production environment.
>>>>> Depending on a distributed file system makes the testing, demo, and
>> local
>>>>> setup harder than init containers.
>>>>>
>>>>> Best,
>>>>> Jark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>>>>>
>>>>>> The FLIP is in the early phase and some details are not included, but
>>>>>> fortunately, we got lots of valuable ideas from the discussion.
>>>>>>
>>>>>> Thanks to everyone who joined the dissuasion!
>>>>>> @Weihua @Shanmon @Shengkai @Biao @Jark
>>>>>>
>>>>>> This weekend I’m gonna revisit and update the FLIP, adding more
>>>>>> details. Hopefully, we can further align our opinions.
>>>>>>
>>>>>> Best,
>>>>>> Paul Lam
>>>>>>
>>>>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>>> 写道：
>>>>>>>
>>>>>>> Hi Jark,
>>>>>>>
>>>>>>> Thanks a lot for your input!
>>>>>>>
>>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>> still
>>>>>>>> necessary to support SQL Driver?
>>>>>>>
>>>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
>> simplify
>>>>>>> Flink SQL execution with Flink CLI.
>>>>>>>
>>>>>>> As the FLIP said, it’s good to have a default main class for Flink
>>>> SQLs,
>>>>>>> which allows users to submit Flink SQLs in the same way as DataStream
>>>>>>> jobs, or else users need to write their own main class.
>>>>>>>
>>>>>>>> SQL Driver needs to serialize SessionState which is very challenging
>>>>>>>> but not detailed covered in the FLIP.
>>>>>>>
>>>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>>>> SessionState? If not, we could make SQL Driver accepts two serialized
>>>>>>> formats:
>>>>>>>
>>>>>>> - SQL files for user-facing public usage
>>>>>>> - ExecNodeGraph for internal usage
>>>>>>>
>>>>>>> It’s kind of similar to the relationship between job jars and
>>>> jobgraphs.
>>>>>>>
>>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>> true?
>>>>>> Is it
>>>>>>>> possible to support it?
>>>>>>>
>>>>>>> Yes, K8s doesn’t distribute any files. It’s the users’ responsibility
>>>> to
>>>>>> make
>>>>>>> sure the resources are accessible in the containers. The common
>>>> solutions
>>>>>>> I know is to use distributed file systems or use init containers to
>>>>>> localize the
>>>>>>> resources.
>>>>>>>
>>>>>>> Now I lean toward introducing a fs to do the distribution job. WDYT?
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>
>> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>
>>>> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>> <mailto:
>> imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>>>
>>>>>> 写道：
>>>>>>>>
>>>>>>>> Hi Paul,
>>>>>>>>
>>>>>>>> Thanks for starting this discussion. I like the proposal! This is a
>>>>>>>> frequently requested feature!
>>>>>>>>
>>>>>>>> I agree with Shengkai that ExecNodeGraph as the submission object
>> is a
>>>>>>>> better idea than SQL file. To be more specific, it should be
>>>>>> JsonPlanGraph
>>>>>>>> or CompiledPlan which is the serializable representation.
>> CompiledPlan
>>>>>> is a
>>>>>>>> clear separation between compiling/optimization/validation and
>>>>>> execution.
>>>>>>>> This can keep the validation and metadata accessing still on the
>>>>>> SQLGateway
>>>>>>>> side. This allows SQLGateway to leverage some metadata caching and
>> UDF
>>>>>> JAR
>>>>>>>> caching for better compiling performance.
>>>>>>>>
>>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
>> still
>>>>>>>> necessary to support SQL Driver? Regarding non-interactive SQL jobs,
>>>>>> users
>>>>>>>> can use the Table API program for application mode. SQL Driver needs
>>>> to
>>>>>>>> serialize SessionState which is very challenging but not detailed
>>>>>> covered
>>>>>>>> in the FLIP.
>>>>>>>>
>>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
>> true?
>>>>>> Is it
>>>>>>>> possible to support it?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Jark
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>> <mailto:
>>>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Weihua,
>>>>>>>>>
>>>>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
>>>>>> challenging
>>>>>>>>> parts of this FLIP.
>>>>>>>>>
>>>>>>>>> Web submission is not enabled in application mode currently as you
>>>>>> said,
>>>>>>>>> but it could be changed if we have good reasons.
>>>>>>>>>
>>>>>>>>> What do you think about introducing a distributed storage for SQL
>>>>>> Gateway?
>>>>>>>>>
>>>>>>>>> We could make use of Flink file systems [1] to distribute the SQL
>>>>>> Gateway
>>>>>>>>> generated resources, that should solve the problem at its root
>> cause.
>>>>>>>>>
>>>>>>>>> Users could specify Flink-supported file systems to ship files.
>> It’s
>>>>>> only
>>>>>>>>> required when using SQL Gateway with K8s application mode.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>>
>>>>>>
>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>> <
>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>>> <
>>>>>>
>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>> <
>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Paul Lam
>>>>>>>>>
>>>>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
>> huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
>>>> huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:huweihua.ckl@gmail.com <ma...@gmail.com>>>> 写道：
>>>>>>>>>>
>>>>>>>>>> Thanks Paul for your reply.
>>>>>>>>>>
>>>>>>>>>> SQLDriver looks good to me.
>>>>>>>>>>
>>>>>>>>>> 2. Do you mean a pass the SQL string a configuration or a program
>>>>>>>>> argument?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I brought this up because we were unable to pass the SQL file to
>>>> Flink
>>>>>>>>>> using Kubernetes mode.
>>>>>>>>>> For DataStream/Python users, they need to prepare their images for
>>>> the
>>>>>>>>> jars
>>>>>>>>>> and dependencies.
>>>>>>>>>> But for SQL users, they can use a common image to run different
>> SQL
>>>>>>>>> queries
>>>>>>>>>> if there are no other udf requirements.
>>>>>>>>>> It would be great if the SQL query and image were not bound.
>>>>>>>>>>
>>>>>>>>>> Using strings is a way to decouple these, but just as you
>> mentioned,
>>>>>> it's
>>>>>>>>>> not easy to pass complex SQL.
>>>>>>>>>>
>>>>>>>>>>> use web submission
>>>>>>>>>> AFAIK, we can not use web submission in the Application mode.
>> Please
>>>>>>>>>> correct me if I'm wrong.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Weihua
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>
>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Biao,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your comments!
>>>>>>>>>>>
>>>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink
>> SQL
>>>>>> jobs
>>>>>>>>>>> in
>>>>>>>>>>>> Application mode? More specifically, if we use SQL
>> client/gateway
>>>> to
>>>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
>>>> flink
>>>>>> to
>>>>>>>>>>> use
>>>>>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>>>>>
>>>>>>>>>>> Thanks for pointing it out. I think only DMLs would be executed
>> via
>>>>>> SQL
>>>>>>>>>>> Driver.
>>>>>>>>>>> I'll add the scope to the FLIP.
>>>>>>>>>>>
>>>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>> trivial as
>>>>>>>>>>> we
>>>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
>>>>>> more
>>>>>>>>>>>> complicated as Shengkai said.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Your input is very informative. I’m thinking about using web
>>>>>> submission,
>>>>>>>>>>> but it requires exposing the JobManager port which could also be
>> a
>>>>>>>>> problem
>>>>>>>>>>> on K8s.
>>>>>>>>>>>
>>>>>>>>>>> Another approach is to explicitly require a distributed storage
>> to
>>>>>> ship
>>>>>>>>>>> files,
>>>>>>>>>>> but we may need a new deployment executor for that.
>>>>>>>>>>>
>>>>>>>>>>> What do you think of these two approaches?
>>>>>>>>>>>
>>>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
>> some
>>>>>>>>>>>> unserializable fields
>>>>>>>>>>>> like
>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>>>> It
>>>>>>>>>>>> may be worthwhile to add more details about the serialization
>>>> part.
>>>>>>>>>>>
>>>>>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as
>>>>>> Shengkai
>>>>>>>>>>> mentioned, do we eliminate the need for serialization of
>>>>>> SessionState?
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Paul Lam
>>>>>>>>>>>
>>>>>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <ma...@gmail.com> <mailto:
>> biaogeng7@gmail.com <ma...@gmail.com>> <mailto:
>>>> biaogeng7@gmail.com <ma...@gmail.com> <mailto:biaogeng7@gmail.com <ma...@gmail.com>>>> 写道：
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Paul for the proposal!I believe it would be very useful
>> for
>>>>>>>>> flink
>>>>>>>>>>>> users.
>>>>>>>>>>>> After reading the FLIP, I have some questions:
>>>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink
>> SQL
>>>>>> jobs
>>>>>>>>>>> in
>>>>>>>>>>>> Application mode? More specifically, if we use SQL
>> client/gateway
>>>> to
>>>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
>>>> flink
>>>>>> to
>>>>>>>>>>> use
>>>>>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>>>> trivial as
>>>>>>>>>>> we
>>>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
>>>>>> more
>>>>>>>>>>>> complicated as Shengkai said. I have implemented a simple POC
>>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> <
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>>>
>>>> <
>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> <
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>>>
>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> based on SQL client before(i.e. consider the SQL client which
>>>>>> supports
>>>>>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One
>> problem
>>>> I
>>>>>>>>> have
>>>>>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
>>>>>>>>> Without
>>>>>>>>>>>> such support, users have to modify the initContainer or rebuild
>> a
>>>>>> new
>>>>>>>>> K8s
>>>>>>>>>>>> image every time to fetch the SQL file. Like the flink k8s
>>>> operator,
>>>>>>>>> one
>>>>>>>>>>>> workaround is to utilize the flink config(transforming the SQL
>>>> file
>>>>>> to
>>>>>>>>> a
>>>>>>>>>>>> escaped string like Weihua mentioned) which will be converted
>> to a
>>>>>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than
>> 1MB
>>>>>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>> <
>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>>>>).
>>>>>> Not
>>>>>>>>>>> sure
>>>>>>>>>>>> if we have better solutions.
>>>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
>> some
>>>>>>>>>>>> unserializable fields
>>>>>>>>>>>> like
>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>>>> It
>>>>>>>>>>>> may be worthwhile to add more details about the serialization
>>>> part.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Biao Geng
>>>>>>>>>>>>
>>>>>>>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
>>>> 于2023年5月31日周三 11:49写道：
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Weihua,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job.
>>>> (Not
>>>>>>>>>>>>> strong,
>>>>>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
>>>>>> following
>>>>>>>>>>>>> reasons FYI:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>>>>>>>>>> 2. Flink program's main class is sort of like Driver in JDBC
>>>> which
>>>>>>>>>>>>> translates SQLs into
>>>>>>>>>>>>> databases specific languages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we
>> need
>>>>>> to
>>>>>>>>>>>>> prepare
>>>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode, which
>>>> may
>>>>>> be
>>>>>>>>> a
>>>>>>>>>>>>> bit
>>>>>>>>>>>>>> cumbersome.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you mean a pass the SQL string a configuration or a program
>>>>>>>>> argument?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I thought it might be convenient for testing propose, but not
>>>>>>>>>>> recommended
>>>>>>>>>>>>> for production,
>>>>>>>>>>>>> cause Flink SQLs could be complicated and involves lots of
>>>>>> characters
>>>>>>>>>>> that
>>>>>>>>>>>>> need to escape.
>>>>>>>>>>>>>
>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>
>>>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>>>>>> "run-application"
>>>>>>>>>>>>>> command. Does that mean we need to perform automatic detection
>>>> in
>>>>>>>>>>> Flink?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes! It’s like running a PyFlink job with the following
>> command:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ```
>>>>>>>>>>>>> ./bin/flink run \
>>>>>>>>>>>>> --pyModule table.word_count \
>>>>>>>>>>>>> --pyFiles examples/python/table
>>>>>>>>>>>>> ```
>>>>>>>>>>>>>
>>>>>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
>> Driver
>>>>>>>>>>>>> automatically.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> <
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>>>
>>>> <
>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> <
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>>>
>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Paul Lam
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
>> huweihua.ckl@gmail.com <ma...@gmail.com>> <mailto:
>>>> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Paul for the proposal.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have a few questions.
>>>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job.
>>>> (Not
>>>>>>>>>>>>> strong,
>>>>>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we
>> need
>>>>>> to
>>>>>>>>>>>>> prepare
>>>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode, which
>>>> may
>>>>>> be
>>>>>>>>> a
>>>>>>>>>>>>> bit
>>>>>>>>>>>>>> cumbersome.
>>>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>>>>>> "run-application"
>>>>>>>>>>>>>> command. Does that mean we need to perform automatic detection
>>>> in
>>>>>>>>>>> Flink?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Weihua
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
>> paullin3280@gmail.com <ma...@gmail.com>
>>>> <mailto:paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi team,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
>>>>>> introduces
>>>>>>>>> a
>>>>>>>>>>>>> SQL
>>>>>>>>>>>>>>> driver as the
>>>>>>>>>>>>>>> default main class for Flink SQL jobs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Currently, Flink SQL could be executed out of the box either
>>>> via
>>>>>> SQL
>>>>>>>>>>>>>>> Client/Gateway
>>>>>>>>>>>>>>> or embedded in a Flink Java/Python program.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, each one has its drawback:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application
>> deployment
>>>>>> mode
>>>>>>>>>>> [2]
>>>>>>>>>>>>>>> - Flink Java/Python program requires extra work to write a
>>>>>> non-SQL
>>>>>>>>>>>>> program
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the
>> default
>>>>>> main
>>>>>>>>>>>>> class
>>>>>>>>>>>>>>> for SQL jobs.
>>>>>>>>>>>>>>> Please see the FLIP docs for details and feel free to
>> comment.
>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Paul Lam
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Weihua Hu <hu...@gmail.com>.
Hi,
Thanks for updating the FLIP.
I have two cents on the distribution of SQLs and resources.
1. Should we support a common file distribution mechanism for k8s
application mode?
I have seen some issues and requirements on the mailing list.
In our production environment, we implement the download command in the
CliFrontend.
And automatically add an init container to the POD for file downloading.
The advantage of this
is that we can use all Flink-supported file systems to store files.
This need more discussion. I would appreciate hearing more opinions.
2. In this FLIP, we distribute files in two different ways in YARN and
Kubernetes. Can we combine it in one way?
If we don't want to implement a common file distribution for k8s
application mode. Could we use the SQLDriver
to download the files both in YARN and K8S? IMO, this can reduce the cost
of code maintenance.
Best,
Weihua
On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <pa...@gmail.com> wrote:
> Hi Mason,
>
> Thanks for your input!
>
> > +1 for init containers or a more generalized way of obtaining arbitrary
> > files. File fetching isn't specific to just SQL--it also matters for Java
> > applications if the user doesn't want to rebuild a Flink image and just
> > wants to modify the user application fat jar.
>
> I agree that utilizing SQL Drivers in Java applications is equally
> important
> as employing them in SQL Gateway. WRT init containers, I think most
> users use them just as a workaround. For example, wget a jar from the
> maven repo.
>
> We could implement the functionality in SQL Driver in a more graceful
> way and the flink-supported filesystem approach seems to be a
> good choice.
>
> > Also, what do you think about prefixing the config options with
> > `sql-driver` instead of just `sql` to be more specific?
>
> LGTM, since SQL Driver is a public interface and the options are
> specific to it.
>
> Best,
> Paul Lam
>
> > 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
> >
> > Hi Paul,
> >
> > +1 for this feature and supporting SQL file + JSON plans. We get a lot of
> > requests to just be able to submit a SQL file, but the JSON plan
> > optimizations make sense.
> >
> > +1 for init containers or a more generalized way of obtaining arbitrary
> > files. File fetching isn't specific to just SQL--it also matters for Java
> > applications if the user doesn't want to rebuild a Flink image and just
> > wants to modify the user application fat jar.
> >
> > Please note that we could reuse the checkpoint storage like S3/HDFS,
> which
> >> should
> >
> > be required to run Flink in production, so I guess that would be
> acceptable
> >> for most
> >
> > users. WDYT?
> >
> >
> > If you do go this route, it would be nice to support writing these files
> to
> > S3/HDFS via Flink. This makes access control and policy management
> simpler.
> >
> > Also, what do you think about prefixing the config options with
> > `sql-driver` instead of just `sql` to be more specific?
> >
> > Best,
> > Mason
> >
> > On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com>> wrote:
> >
> >> Hi Jark,
> >>
> >> Thanks for your input! Please see my comments inline.
> >>
> >>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> >>> DataStream API also doesn't provide a default main class for users,
> >>> why do we need to provide such one for SQL?
> >>
> >> Sorry for the confusion I caused. By DataStream jobs, I mean jobs
> submitted
> >> via Flink CLI which actually could be DataStream/Table jobs.
> >>
> >> I think a default main class would be user-friendly which eliminates the
> >> need
> >> for users to write a main class as SQLRunner in Flink K8s operator [1].
> >>
> >>> I thought the proposed SqlDriver was a dedicated main class accepting
> >> SQL files, is
> >>> that correct?
> >>
> >> Both JSON plans and SQL files are accepted. SQL Gateway should use JSON
> >> plans,
> >> while CLI users may use either JSON plans or SQL files.
> >>
> >> Please see the updated FLIP[2] for more details.
> >>
> >>> Personally, I prefer the way of init containers which doesn't depend on
> >>> additional components.
> >>> This can reduce the moving parts of a production environment.
> >>> Depending on a distributed file system makes the testing, demo, and
> local
> >>> setup harder than init containers.
> >>
> >> Please note that we could reuse the checkpoint storage like S3/HDFS,
> which
> >> should
> >> be required to run Flink in production, so I guess that would be
> >> acceptable for most
> >> users. WDYT?
> >>
> >> WRT testing, demo, and local setups, I think we could support the local
> >> filesystem
> >> scheme i.e. file://** as the state backends do. It works as long as SQL
> >> Gateway
> >> and JobManager(or SQL Driver) can access the resource directory
> (specified
> >> via
> >> `sql-gateway.application.storage-dir`).
> >>
> >> Thanks!
> >>
> >> [1]
> >>
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> >> [2]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >> [3]
> >>
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
> >>
> >> Best,
> >> Paul Lam
> >>
> >>> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
> >>>
> >>> Hi Paul,
> >>>
> >>> Thanks for your reply. I left my comments inline.
> >>>
> >>>> As the FLIP said, it’s good to have a default main class for Flink
> SQLs,
> >>>> which allows users to submit Flink SQLs in the same way as DataStream
> >>>> jobs, or else users need to write their own main class.
> >>>
> >>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> >>> DataStream API also doesn't provide a default main class for users,
> >>> why do we need to provide such one for SQL?
> >>>
> >>>> With the help of ExecNodeGraph, do we still need the serialized
> >>>> SessionState? If not, we could make SQL Driver accepts two serialized
> >>>> formats:
> >>>
> >>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought the
> >>> proposed SqlDriver was a dedicated main class accepting SQL files, is
> >>> that correct?
> >>> If true, we have to ship the SessionState for this case which is a
> large
> >>> work.
> >>> I think we just need a JsonPlanDriver which is a main class that
> accepts
> >>> JsonPlan as the parameter.
> >>>
> >>>
> >>>> The common solutions I know is to use distributed file systems or use
> >>>> init containers to localize the resources.
> >>>
> >>> Personally, I prefer the way of init containers which doesn't depend on
> >>> additional components.
> >>> This can reduce the moving parts of a production environment.
> >>> Depending on a distributed file system makes the testing, demo, and
> local
> >>> setup harder than init containers.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com> <mailto:
> >> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
> >>>
> >>>> The FLIP is in the early phase and some details are not included, but
> >>>> fortunately, we got lots of valuable ideas from the discussion.
> >>>>
> >>>> Thanks to everyone who joined the dissuasion!
> >>>> @Weihua @Shanmon @Shengkai @Biao @Jark
> >>>>
> >>>> This weekend I’m gonna revisit and update the FLIP, adding more
> >>>> details. Hopefully, we can further align our opinions.
> >>>>
> >>>> Best,
> >>>> Paul Lam
> >>>>
> >>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com>> 写道：
> >>>>>
> >>>>> Hi Jark,
> >>>>>
> >>>>> Thanks a lot for your input!
> >>>>>
> >>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> still
> >>>>>> necessary to support SQL Driver?
> >>>>>
> >>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could
> simplify
> >>>>> Flink SQL execution with Flink CLI.
> >>>>>
> >>>>> As the FLIP said, it’s good to have a default main class for Flink
> >> SQLs,
> >>>>> which allows users to submit Flink SQLs in the same way as DataStream
> >>>>> jobs, or else users need to write their own main class.
> >>>>>
> >>>>>> SQL Driver needs to serialize SessionState which is very challenging
> >>>>>> but not detailed covered in the FLIP.
> >>>>>
> >>>>> With the help of ExecNodeGraph, do we still need the serialized
> >>>>> SessionState? If not, we could make SQL Driver accepts two serialized
> >>>>> formats:
> >>>>>
> >>>>> - SQL files for user-facing public usage
> >>>>> - ExecNodeGraph for internal usage
> >>>>>
> >>>>> It’s kind of similar to the relationship between job jars and
> >> jobgraphs.
> >>>>>
> >>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> true?
> >>>> Is it
> >>>>>> possible to support it?
> >>>>>
> >>>>> Yes, K8s doesn’t distribute any files. It’s the users’ responsibility
> >> to
> >>>> make
> >>>>> sure the resources are accessible in the containers. The common
> >> solutions
> >>>>> I know is to use distributed file systems or use init containers to
> >>>> localize the
> >>>>> resources.
> >>>>>
> >>>>> Now I lean toward introducing a fs to do the distribution job. WDYT?
> >>>>>
> >>>>> Best,
> >>>>> Paul Lam
> >>>>>
> >>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com>
> <mailto:imjark@gmail.com <ma...@gmail.com>>
> >> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:
> imjark@gmail.com <ma...@gmail.com>>>>
> >>>> 写道：
> >>>>>>
> >>>>>> Hi Paul,
> >>>>>>
> >>>>>> Thanks for starting this discussion. I like the proposal! This is a
> >>>>>> frequently requested feature!
> >>>>>>
> >>>>>> I agree with Shengkai that ExecNodeGraph as the submission object
> is a
> >>>>>> better idea than SQL file. To be more specific, it should be
> >>>> JsonPlanGraph
> >>>>>> or CompiledPlan which is the serializable representation.
> CompiledPlan
> >>>> is a
> >>>>>> clear separation between compiling/optimization/validation and
> >>>> execution.
> >>>>>> This can keep the validation and metadata accessing still on the
> >>>> SQLGateway
> >>>>>> side. This allows SQLGateway to leverage some metadata caching and
> UDF
> >>>> JAR
> >>>>>> caching for better compiling performance.
> >>>>>>
> >>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it
> still
> >>>>>> necessary to support SQL Driver? Regarding non-interactive SQL jobs,
> >>>> users
> >>>>>> can use the Table API program for application mode. SQL Driver needs
> >> to
> >>>>>> serialize SessionState which is very challenging but not detailed
> >>>> covered
> >>>>>> in the FLIP.
> >>>>>>
> >>>>>> Regarding "K8S doesn't support shipping multiple jars", is that
> true?
> >>>> Is it
> >>>>>> possible to support it?
> >>>>>>
> >>>>>> Best,
> >>>>>> Jark
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com> <mailto:
> >> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
> >>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:
> paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
> >>>>>>
> >>>>>>> Hi Weihua,
> >>>>>>>
> >>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
> >>>> challenging
> >>>>>>> parts of this FLIP.
> >>>>>>>
> >>>>>>> Web submission is not enabled in application mode currently as you
> >>>> said,
> >>>>>>> but it could be changed if we have good reasons.
> >>>>>>>
> >>>>>>> What do you think about introducing a distributed storage for SQL
> >>>> Gateway?
> >>>>>>>
> >>>>>>> We could make use of Flink file systems [1] to distribute the SQL
> >>>> Gateway
> >>>>>>> generated resources, that should solve the problem at its root
> cause.
> >>>>>>>
> >>>>>>> Users could specify Flink-supported file systems to ship files.
> It’s
> >>>> only
> >>>>>>> required when using SQL Gateway with K8s application mode.
> >>>>>>>
> >>>>>>> [1]
> >>>>>>>
> >>>>
> >>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> <
> >>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >>>
> >>>> <
> >>>>
> >>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> <
> >>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >>>
> >>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> huweihua.ckl@gmail.com> <mailto:
> >> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
> >>>>>>>>
> >>>>>>>> Thanks Paul for your reply.
> >>>>>>>>
> >>>>>>>> SQLDriver looks good to me.
> >>>>>>>>
> >>>>>>>> 2. Do you mean a pass the SQL string a configuration or a program
> >>>>>>> argument?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I brought this up because we were unable to pass the SQL file to
> >> Flink
> >>>>>>>> using Kubernetes mode.
> >>>>>>>> For DataStream/Python users, they need to prepare their images for
> >> the
> >>>>>>> jars
> >>>>>>>> and dependencies.
> >>>>>>>> But for SQL users, they can use a common image to run different
> SQL
> >>>>>>> queries
> >>>>>>>> if there are no other udf requirements.
> >>>>>>>> It would be great if the SQL query and image were not bound.
> >>>>>>>>
> >>>>>>>> Using strings is a way to decouple these, but just as you
> mentioned,
> >>>> it's
> >>>>>>>> not easy to pass complex SQL.
> >>>>>>>>
> >>>>>>>>> use web submission
> >>>>>>>> AFAIK, we can not use web submission in the Application mode.
> Please
> >>>>>>>> correct me if I'm wrong.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Weihua
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com>
> >> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Biao,
> >>>>>>>>>
> >>>>>>>>> Thanks for your comments!
> >>>>>>>>>
> >>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink
> SQL
> >>>> jobs
> >>>>>>>>> in
> >>>>>>>>>> Application mode? More specifically, if we use SQL
> client/gateway
> >> to
> >>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
> >> flink
> >>>> to
> >>>>>>>>> use
> >>>>>>>>>> Application mode to execute those queries after this FLIP?
> >>>>>>>>>
> >>>>>>>>> Thanks for pointing it out. I think only DMLs would be executed
> via
> >>>> SQL
> >>>>>>>>> Driver.
> >>>>>>>>> I'll add the scope to the FLIP.
> >>>>>>>>>
> >>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> >>>> trivial as
> >>>>>>>>> we
> >>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
> >>>> more
> >>>>>>>>>> complicated as Shengkai said.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Your input is very informative. I’m thinking about using web
> >>>> submission,
> >>>>>>>>> but it requires exposing the JobManager port which could also be
> a
> >>>>>>> problem
> >>>>>>>>> on K8s.
> >>>>>>>>>
> >>>>>>>>> Another approach is to explicitly require a distributed storage
> to
> >>>> ship
> >>>>>>>>> files,
> >>>>>>>>> but we may need a new deployment executor for that.
> >>>>>>>>>
> >>>>>>>>> What do you think of these two approaches?
> >>>>>>>>>
> >>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
> some
> >>>>>>>>>> unserializable fields
> >>>>>>>>>> like
> >>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >>>>>>> It
> >>>>>>>>>> may be worthwhile to add more details about the serialization
> >> part.
> >>>>>>>>>
> >>>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as
> >>>> Shengkai
> >>>>>>>>> mentioned, do we eliminate the need for serialization of
> >>>> SessionState?
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Paul Lam
> >>>>>>>>>
> >>>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
> biaogeng7@gmail.com> <mailto:
> >> biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
> >>>>>>>>>>
> >>>>>>>>>> Thanks Paul for the proposal!I believe it would be very useful
> for
> >>>>>>> flink
> >>>>>>>>>> users.
> >>>>>>>>>> After reading the FLIP, I have some questions:
> >>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink
> SQL
> >>>> jobs
> >>>>>>>>> in
> >>>>>>>>>> Application mode? More specifically, if we use SQL
> client/gateway
> >> to
> >>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
> >> flink
> >>>> to
> >>>>>>>>> use
> >>>>>>>>>> Application mode to execute those queries after this FLIP?
> >>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> >>>> trivial as
> >>>>>>>>> we
> >>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
> >>>> more
> >>>>>>>>>> complicated as Shengkai said. I have implemented a simple POC
> >>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> <
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >> <
> >>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> <
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >>>
> >>>>>>>>>>
> >>>>>>>>>> based on SQL client before(i.e. consider the SQL client which
> >>>> supports
> >>>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One
> problem
> >> I
> >>>>>>> have
> >>>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
> >>>>>>> Without
> >>>>>>>>>> such support, users have to modify the initContainer or rebuild
> a
> >>>> new
> >>>>>>> K8s
> >>>>>>>>>> image every time to fetch the SQL file. Like the flink k8s
> >> operator,
> >>>>>>> one
> >>>>>>>>>> workaround is to utilize the flink config(transforming the SQL
> >> file
> >>>> to
> >>>>>>> a
> >>>>>>>>>> escaped string like Weihua mentioned) which will be converted
> to a
> >>>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than
> 1MB
> >>>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/ <
> https://kubernetes.io/docs/concepts/configuration/configmap/> <
> >> https://kubernetes.io/docs/concepts/configuration/configmap/ <
> https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
> >>>> Not
> >>>>>>>>> sure
> >>>>>>>>>> if we have better solutions.
> >>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are
> some
> >>>>>>>>>> unserializable fields
> >>>>>>>>>> like
> >>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >>>>>>> It
> >>>>>>>>>> may be worthwhile to add more details about the serialization
> >> part.
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Biao Geng
> >>>>>>>>>>
> >>>>>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> >> 于2023年5月31日周三 11:49写道：
> >>>>>>>>>>
> >>>>>>>>>>> Hi Weihua,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks a lot for your input! Please see my comments inline.
> >>>>>>>>>>>
> >>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job.
> >> (Not
> >>>>>>>>>>> strong,
> >>>>>>>>>>>> the SQLDriver is fine for me)
> >>>>>>>>>>>
> >>>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
> >>>> following
> >>>>>>>>>>> reasons FYI:
> >>>>>>>>>>>
> >>>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
> >>>>>>>>>>> 2. Flink program's main class is sort of like Driver in JDBC
> >> which
> >>>>>>>>>>> translates SQLs into
> >>>>>>>>>>> databases specific languages.
> >>>>>>>>>>>
> >>>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >>>>>>>>>>>
> >>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we
> need
> >>>> to
> >>>>>>>>>>> prepare
> >>>>>>>>>>>> a SQL file in an image for Kubernetes application mode, which
> >> may
> >>>> be
> >>>>>>> a
> >>>>>>>>>>> bit
> >>>>>>>>>>>> cumbersome.
> >>>>>>>>>>>
> >>>>>>>>>>> Do you mean a pass the SQL string a configuration or a program
> >>>>>>> argument?
> >>>>>>>>>>>
> >>>>>>>>>>> I thought it might be convenient for testing propose, but not
> >>>>>>>>> recommended
> >>>>>>>>>>> for production,
> >>>>>>>>>>> cause Flink SQLs could be complicated and involves lots of
> >>>> characters
> >>>>>>>>> that
> >>>>>>>>>>> need to escape.
> >>>>>>>>>>>
> >>>>>>>>>>> WDYT?
> >>>>>>>>>>>
> >>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>>>>>>>>> "run-application"
> >>>>>>>>>>>> command. Does that mean we need to perform automatic detection
> >> in
> >>>>>>>>> Flink?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes! It’s like running a PyFlink job with the following
> command:
> >>>>>>>>>>>
> >>>>>>>>>>> ```
> >>>>>>>>>>> ./bin/flink run \
> >>>>>>>>>>> --pyModule table.word_count \
> >>>>>>>>>>> --pyFiles examples/python/table
> >>>>>>>>>>> ```
> >>>>>>>>>>>
> >>>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL
> Driver
> >>>>>>>>>>> automatically.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> <
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >> <
> >>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> <
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >>>
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Paul Lam
> >>>>>>>>>>>
> >>>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> huweihua.ckl@gmail.com> <mailto:
> >> huweihua.ckl@gmail.com>> 写道：
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks Paul for the proposal.
> >>>>>>>>>>>>
> >>>>>>>>>>>> +1 for this. It is valuable in improving ease of use.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have a few questions.
> >>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job.
> >> (Not
> >>>>>>>>>>> strong,
> >>>>>>>>>>>> the SQLDriver is fine for me)
> >>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we
> need
> >>>> to
> >>>>>>>>>>> prepare
> >>>>>>>>>>>> a SQL file in an image for Kubernetes application mode, which
> >> may
> >>>> be
> >>>>>>> a
> >>>>>>>>>>> bit
> >>>>>>>>>>>> cumbersome.
> >>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>>>>>>>>> "run-application"
> >>>>>>>>>>>> command. Does that mean we need to perform automatic detection
> >> in
> >>>>>>>>> Flink?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Weihua
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <
> paullin3280@gmail.com
> >> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi team,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
> >>>> introduces
> >>>>>>> a
> >>>>>>>>>>> SQL
> >>>>>>>>>>>>> driver as the
> >>>>>>>>>>>>> default main class for Flink SQL jobs.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Currently, Flink SQL could be executed out of the box either
> >> via
> >>>> SQL
> >>>>>>>>>>>>> Client/Gateway
> >>>>>>>>>>>>> or embedded in a Flink Java/Python program.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> However, each one has its drawback:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application
> deployment
> >>>> mode
> >>>>>>>>> [2]
> >>>>>>>>>>>>> - Flink Java/Python program requires extra work to write a
> >>>> non-SQL
> >>>>>>>>>>> program
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the
> default
> >>>> main
> >>>>>>>>>>> class
> >>>>>>>>>>>>> for SQL jobs.
> >>>>>>>>>>>>> Please see the FLIP docs for details and feel free to
> comment.
> >>>>>>> Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >>>>>>>>>>>>> <
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Paul Lam
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Mason,
Thanks for your input!
> +1 for init containers or a more generalized way of obtaining arbitrary
> files. File fetching isn't specific to just SQL--it also matters for Java
> applications if the user doesn't want to rebuild a Flink image and just
> wants to modify the user application fat jar.
I agree that utilizing SQL Drivers in Java applications is equally important
as employing them in SQL Gateway. WRT init containers, I think most
users use them just as a workaround. For example, wget a jar from the
maven repo.
We could implement the functionality in SQL Driver in a more graceful
way and the flink-supported filesystem approach seems to be a
good choice.
> Also, what do you think about prefixing the config options with
> `sql-driver` instead of just `sql` to be more specific?
LGTM, since SQL Driver is a public interface and the options are
specific to it.
Best,
Paul Lam
> 2023年6月6日 06:30，Mason Chen <ma...@gmail.com> 写道：
>
> Hi Paul,
>
> +1 for this feature and supporting SQL file + JSON plans. We get a lot of
> requests to just be able to submit a SQL file, but the JSON plan
> optimizations make sense.
>
> +1 for init containers or a more generalized way of obtaining arbitrary
> files. File fetching isn't specific to just SQL--it also matters for Java
> applications if the user doesn't want to rebuild a Flink image and just
> wants to modify the user application fat jar.
>
> Please note that we could reuse the checkpoint storage like S3/HDFS, which
>> should
>
> be required to run Flink in production, so I guess that would be acceptable
>> for most
>
> users. WDYT?
>
>
> If you do go this route, it would be nice to support writing these files to
> S3/HDFS via Flink. This makes access control and policy management simpler.
>
> Also, what do you think about prefixing the config options with
> `sql-driver` instead of just `sql` to be more specific?
>
> Best,
> Mason
>
> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
>
>> Hi Jark,
>>
>> Thanks for your input! Please see my comments inline.
>>
>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>> DataStream API also doesn't provide a default main class for users,
>>> why do we need to provide such one for SQL?
>>
>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs submitted
>> via Flink CLI which actually could be DataStream/Table jobs.
>>
>> I think a default main class would be user-friendly which eliminates the
>> need
>> for users to write a main class as SQLRunner in Flink K8s operator [1].
>>
>>> I thought the proposed SqlDriver was a dedicated main class accepting
>> SQL files, is
>>> that correct?
>>
>> Both JSON plans and SQL files are accepted. SQL Gateway should use JSON
>> plans,
>> while CLI users may use either JSON plans or SQL files.
>>
>> Please see the updated FLIP[2] for more details.
>>
>>> Personally, I prefer the way of init containers which doesn't depend on
>>> additional components.
>>> This can reduce the moving parts of a production environment.
>>> Depending on a distributed file system makes the testing, demo, and local
>>> setup harder than init containers.
>>
>> Please note that we could reuse the checkpoint storage like S3/HDFS, which
>> should
>> be required to run Flink in production, so I guess that would be
>> acceptable for most
>> users. WDYT?
>>
>> WRT testing, demo, and local setups, I think we could support the local
>> filesystem
>> scheme i.e. file://** as the state backends do. It works as long as SQL
>> Gateway
>> and JobManager(or SQL Driver) can access the resource directory (specified
>> via
>> `sql-gateway.application.storage-dir`).
>>
>> Thanks!
>>
>> [1]
>> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
>> [2]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>> [3]
>> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>>
>> Best,
>> Paul Lam
>>
>>> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
>>>
>>> Hi Paul,
>>>
>>> Thanks for your reply. I left my comments inline.
>>>
>>>> As the FLIP said, it’s good to have a default main class for Flink SQLs,
>>>> which allows users to submit Flink SQLs in the same way as DataStream
>>>> jobs, or else users need to write their own main class.
>>>
>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
>>> DataStream API also doesn't provide a default main class for users,
>>> why do we need to provide such one for SQL?
>>>
>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>> SessionState? If not, we could make SQL Driver accepts two serialized
>>>> formats:
>>>
>>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought the
>>> proposed SqlDriver was a dedicated main class accepting SQL files, is
>>> that correct?
>>> If true, we have to ship the SessionState for this case which is a large
>>> work.
>>> I think we just need a JsonPlanDriver which is a main class that accepts
>>> JsonPlan as the parameter.
>>>
>>>
>>>> The common solutions I know is to use distributed file systems or use
>>>> init containers to localize the resources.
>>>
>>> Personally, I prefer the way of init containers which doesn't depend on
>>> additional components.
>>> This can reduce the moving parts of a production environment.
>>> Depending on a distributed file system makes the testing, demo, and local
>>> setup harder than init containers.
>>>
>>> Best,
>>> Jark
>>>
>>>
>>>
>>>
>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>>>
>>>> The FLIP is in the early phase and some details are not included, but
>>>> fortunately, we got lots of valuable ideas from the discussion.
>>>>
>>>> Thanks to everyone who joined the dissuasion!
>>>> @Weihua @Shanmon @Shengkai @Biao @Jark
>>>>
>>>> This weekend I’m gonna revisit and update the FLIP, adding more
>>>> details. Hopefully, we can further align our opinions.
>>>>
>>>> Best,
>>>> Paul Lam
>>>>
>>>>> 2023年6月2日 18:02，Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> 写道：
>>>>>
>>>>> Hi Jark,
>>>>>
>>>>> Thanks a lot for your input!
>>>>>
>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it still
>>>>>> necessary to support SQL Driver?
>>>>>
>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could simplify
>>>>> Flink SQL execution with Flink CLI.
>>>>>
>>>>> As the FLIP said, it’s good to have a default main class for Flink
>> SQLs,
>>>>> which allows users to submit Flink SQLs in the same way as DataStream
>>>>> jobs, or else users need to write their own main class.
>>>>>
>>>>>> SQL Driver needs to serialize SessionState which is very challenging
>>>>>> but not detailed covered in the FLIP.
>>>>>
>>>>> With the help of ExecNodeGraph, do we still need the serialized
>>>>> SessionState? If not, we could make SQL Driver accepts two serialized
>>>>> formats:
>>>>>
>>>>> - SQL files for user-facing public usage
>>>>> - ExecNodeGraph for internal usage
>>>>>
>>>>> It’s kind of similar to the relationship between job jars and
>> jobgraphs.
>>>>>
>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that true?
>>>> Is it
>>>>>> possible to support it?
>>>>>
>>>>> Yes, K8s doesn’t distribute any files. It’s the users’ responsibility
>> to
>>>> make
>>>>> sure the resources are accessible in the containers. The common
>> solutions
>>>>> I know is to use distributed file systems or use init containers to
>>>> localize the
>>>>> resources.
>>>>>
>>>>> Now I lean toward introducing a fs to do the distribution job. WDYT?
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>
>> <mailto:imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>>
>>>> 写道：
>>>>>>
>>>>>> Hi Paul,
>>>>>>
>>>>>> Thanks for starting this discussion. I like the proposal! This is a
>>>>>> frequently requested feature!
>>>>>>
>>>>>> I agree with Shengkai that ExecNodeGraph as the submission object is a
>>>>>> better idea than SQL file. To be more specific, it should be
>>>> JsonPlanGraph
>>>>>> or CompiledPlan which is the serializable representation. CompiledPlan
>>>> is a
>>>>>> clear separation between compiling/optimization/validation and
>>>> execution.
>>>>>> This can keep the validation and metadata accessing still on the
>>>> SQLGateway
>>>>>> side. This allows SQLGateway to leverage some metadata caching and UDF
>>>> JAR
>>>>>> caching for better compiling performance.
>>>>>>
>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it still
>>>>>> necessary to support SQL Driver? Regarding non-interactive SQL jobs,
>>>> users
>>>>>> can use the Table API program for application mode. SQL Driver needs
>> to
>>>>>> serialize SessionState which is very challenging but not detailed
>>>> covered
>>>>>> in the FLIP.
>>>>>>
>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that true?
>>>> Is it
>>>>>> possible to support it?
>>>>>>
>>>>>> Best,
>>>>>> Jark
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>> <mailto:
>>>> paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>> wrote:
>>>>>>
>>>>>>> Hi Weihua,
>>>>>>>
>>>>>>> You’re right. Distributing the SQLs to the TMs is one of the
>>>> challenging
>>>>>>> parts of this FLIP.
>>>>>>>
>>>>>>> Web submission is not enabled in application mode currently as you
>>>> said,
>>>>>>> but it could be changed if we have good reasons.
>>>>>>>
>>>>>>> What do you think about introducing a distributed storage for SQL
>>>> Gateway?
>>>>>>>
>>>>>>> We could make use of Flink file systems [1] to distribute the SQL
>>>> Gateway
>>>>>>> generated resources, that should solve the problem at its root cause.
>>>>>>>
>>>>>>> Users could specify Flink-supported file systems to ship files. It’s
>>>> only
>>>>>>> required when using SQL Gateway with K8s application mode.
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>> <
>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
>> huweihua.ckl@gmail.com <ma...@gmail.com>>> 写道：
>>>>>>>>
>>>>>>>> Thanks Paul for your reply.
>>>>>>>>
>>>>>>>> SQLDriver looks good to me.
>>>>>>>>
>>>>>>>> 2. Do you mean a pass the SQL string a configuration or a program
>>>>>>> argument?
>>>>>>>>
>>>>>>>>
>>>>>>>> I brought this up because we were unable to pass the SQL file to
>> Flink
>>>>>>>> using Kubernetes mode.
>>>>>>>> For DataStream/Python users, they need to prepare their images for
>> the
>>>>>>> jars
>>>>>>>> and dependencies.
>>>>>>>> But for SQL users, they can use a common image to run different SQL
>>>>>>> queries
>>>>>>>> if there are no other udf requirements.
>>>>>>>> It would be great if the SQL query and image were not bound.
>>>>>>>>
>>>>>>>> Using strings is a way to decouple these, but just as you mentioned,
>>>> it's
>>>>>>>> not easy to pass complex SQL.
>>>>>>>>
>>>>>>>>> use web submission
>>>>>>>> AFAIK, we can not use web submission in the Application mode. Please
>>>>>>>> correct me if I'm wrong.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Weihua
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Biao,
>>>>>>>>>
>>>>>>>>> Thanks for your comments!
>>>>>>>>>
>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL
>>>> jobs
>>>>>>>>> in
>>>>>>>>>> Application mode? More specifically, if we use SQL client/gateway
>> to
>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
>> flink
>>>> to
>>>>>>>>> use
>>>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>>>
>>>>>>>>> Thanks for pointing it out. I think only DMLs would be executed via
>>>> SQL
>>>>>>>>> Driver.
>>>>>>>>> I'll add the scope to the FLIP.
>>>>>>>>>
>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>> trivial as
>>>>>>>>> we
>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
>>>> more
>>>>>>>>>> complicated as Shengkai said.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Your input is very informative. I’m thinking about using web
>>>> submission,
>>>>>>>>> but it requires exposing the JobManager port which could also be a
>>>>>>> problem
>>>>>>>>> on K8s.
>>>>>>>>>
>>>>>>>>> Another approach is to explicitly require a distributed storage to
>>>> ship
>>>>>>>>> files,
>>>>>>>>> but we may need a new deployment executor for that.
>>>>>>>>>
>>>>>>>>> What do you think of these two approaches?
>>>>>>>>>
>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are some
>>>>>>>>>> unserializable fields
>>>>>>>>>> like
>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>> It
>>>>>>>>>> may be worthwhile to add more details about the serialization
>> part.
>>>>>>>>>
>>>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as
>>>> Shengkai
>>>>>>>>> mentioned, do we eliminate the need for serialization of
>>>> SessionState?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Paul Lam
>>>>>>>>>
>>>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <ma...@gmail.com> <mailto:
>> biaogeng7@gmail.com <ma...@gmail.com>>> 写道：
>>>>>>>>>>
>>>>>>>>>> Thanks Paul for the proposal!I believe it would be very useful for
>>>>>>> flink
>>>>>>>>>> users.
>>>>>>>>>> After reading the FLIP, I have some questions:
>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL
>>>> jobs
>>>>>>>>> in
>>>>>>>>>> Application mode? More specifically, if we use SQL client/gateway
>> to
>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
>> flink
>>>> to
>>>>>>>>> use
>>>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>>>> trivial as
>>>>>>>>> we
>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
>>>> more
>>>>>>>>>> complicated as Shengkai said. I have implemented a simple POC
>>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>> <
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>>>
>>>>>>>>>>
>>>>>>>>>> based on SQL client before(i.e. consider the SQL client which
>>>> supports
>>>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One problem
>> I
>>>>>>> have
>>>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
>>>>>>> Without
>>>>>>>>>> such support, users have to modify the initContainer or rebuild a
>>>> new
>>>>>>> K8s
>>>>>>>>>> image every time to fetch the SQL file. Like the flink k8s
>> operator,
>>>>>>> one
>>>>>>>>>> workaround is to utilize the flink config(transforming the SQL
>> file
>>>> to
>>>>>>> a
>>>>>>>>>> escaped string like Weihua mentioned) which will be converted to a
>>>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
>>>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/> <
>> https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>>>).
>>>> Not
>>>>>>>>> sure
>>>>>>>>>> if we have better solutions.
>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are some
>>>>>>>>>> unserializable fields
>>>>>>>>>> like
>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>>>> It
>>>>>>>>>> may be worthwhile to add more details about the serialization
>> part.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Biao Geng
>>>>>>>>>>
>>>>>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>> 于2023年5月31日周三 11:49写道：
>>>>>>>>>>
>>>>>>>>>>> Hi Weihua,
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>>>>>>>
>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job.
>> (Not
>>>>>>>>>>> strong,
>>>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>>>
>>>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
>>>> following
>>>>>>>>>>> reasons FYI:
>>>>>>>>>>>
>>>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>>>>>>>> 2. Flink program's main class is sort of like Driver in JDBC
>> which
>>>>>>>>>>> translates SQLs into
>>>>>>>>>>> databases specific languages.
>>>>>>>>>>>
>>>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>>>>>>>
>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need
>>>> to
>>>>>>>>>>> prepare
>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode, which
>> may
>>>> be
>>>>>>> a
>>>>>>>>>>> bit
>>>>>>>>>>>> cumbersome.
>>>>>>>>>>>
>>>>>>>>>>> Do you mean a pass the SQL string a configuration or a program
>>>>>>> argument?
>>>>>>>>>>>
>>>>>>>>>>> I thought it might be convenient for testing propose, but not
>>>>>>>>> recommended
>>>>>>>>>>> for production,
>>>>>>>>>>> cause Flink SQLs could be complicated and involves lots of
>>>> characters
>>>>>>>>> that
>>>>>>>>>>> need to escape.
>>>>>>>>>>>
>>>>>>>>>>> WDYT?
>>>>>>>>>>>
>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>>>> "run-application"
>>>>>>>>>>>> command. Does that mean we need to perform automatic detection
>> in
>>>>>>>>> Flink?
>>>>>>>>>>>
>>>>>>>>>>> Yes! It’s like running a PyFlink job with the following command:
>>>>>>>>>>>
>>>>>>>>>>> ```
>>>>>>>>>>> ./bin/flink run \
>>>>>>>>>>> --pyModule table.word_count \
>>>>>>>>>>> --pyFiles examples/python/table
>>>>>>>>>>> ```
>>>>>>>>>>>
>>>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
>>>>>>>>>>> automatically.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>> <
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Paul Lam
>>>>>>>>>>>
>>>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com> <mailto:
>> huweihua.ckl@gmail.com>> 写道：
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Paul for the proposal.
>>>>>>>>>>>>
>>>>>>>>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>>>>>>>>
>>>>>>>>>>>> I have a few questions.
>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job.
>> (Not
>>>>>>>>>>> strong,
>>>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need
>>>> to
>>>>>>>>>>> prepare
>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode, which
>> may
>>>> be
>>>>>>> a
>>>>>>>>>>> bit
>>>>>>>>>>>> cumbersome.
>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>>>> "run-application"
>>>>>>>>>>>> command. Does that mean we need to perform automatic detection
>> in
>>>>>>>>> Flink?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Weihua
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <paullin3280@gmail.com
>> <mailto:paullin3280@gmail.com <ma...@gmail.com>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi team,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
>>>> introduces
>>>>>>> a
>>>>>>>>>>> SQL
>>>>>>>>>>>>> driver as the
>>>>>>>>>>>>> default main class for Flink SQL jobs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Currently, Flink SQL could be executed out of the box either
>> via
>>>> SQL
>>>>>>>>>>>>> Client/Gateway
>>>>>>>>>>>>> or embedded in a Flink Java/Python program.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, each one has its drawback:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application deployment
>>>> mode
>>>>>>>>> [2]
>>>>>>>>>>>>> - Flink Java/Python program requires extra work to write a
>>>> non-SQL
>>>>>>>>>>> program
>>>>>>>>>>>>>
>>>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the default
>>>> main
>>>>>>>>>>> class
>>>>>>>>>>>>> for SQL jobs.
>>>>>>>>>>>>> Please see the FLIP docs for details and feel free to comment.
>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>>>>>>>>>> <
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>>>>>>>>>>
>>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Paul Lam
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Mason Chen <ma...@gmail.com>.
Hi Paul,
+1 for this feature and supporting SQL file + JSON plans. We get a lot of
requests to just be able to submit a SQL file, but the JSON plan
optimizations make sense.
+1 for init containers or a more generalized way of obtaining arbitrary
files. File fetching isn't specific to just SQL--it also matters for Java
applications if the user doesn't want to rebuild a Flink image and just
wants to modify the user application fat jar.
Please note that we could reuse the checkpoint storage like S3/HDFS, which
> should
be required to run Flink in production, so I guess that would be acceptable
> for most
users. WDYT?
If you do go this route, it would be nice to support writing these files to
S3/HDFS via Flink. This makes access control and policy management simpler.
Also, what do you think about prefixing the config options with
`sql-driver` instead of just `sql` to be more specific?
Best,
Mason
On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <pa...@gmail.com> wrote:
> Hi Jark,
>
> Thanks for your input! Please see my comments inline.
>
> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > DataStream API also doesn't provide a default main class for users,
> > why do we need to provide such one for SQL?
>
> Sorry for the confusion I caused. By DataStream jobs, I mean jobs submitted
> via Flink CLI which actually could be DataStream/Table jobs.
>
> I think a default main class would be user-friendly which eliminates the
> need
> for users to write a main class as SQLRunner in Flink K8s operator [1].
>
> > I thought the proposed SqlDriver was a dedicated main class accepting
> SQL files, is
> > that correct?
>
> Both JSON plans and SQL files are accepted. SQL Gateway should use JSON
> plans,
> while CLI users may use either JSON plans or SQL files.
>
> Please see the updated FLIP[2] for more details.
>
> > Personally, I prefer the way of init containers which doesn't depend on
> > additional components.
> > This can reduce the moving parts of a production environment.
> > Depending on a distributed file system makes the testing, demo, and local
> > setup harder than init containers.
>
> Please note that we could reuse the checkpoint storage like S3/HDFS, which
> should
> be required to run Flink in production, so I guess that would be
> acceptable for most
> users. WDYT?
>
> WRT testing, demo, and local setups, I think we could support the local
> filesystem
> scheme i.e. file://** as the state backends do. It works as long as SQL
> Gateway
> and JobManager(or SQL Driver) can access the resource directory (specified
> via
> `sql-gateway.application.storage-dir`).
>
> Thanks!
>
> [1]
> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> [3]
> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
>
> Best,
> Paul Lam
>
> > 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
> >
> > Hi Paul,
> >
> > Thanks for your reply. I left my comments inline.
> >
> >> As the FLIP said, it’s good to have a default main class for Flink SQLs,
> >> which allows users to submit Flink SQLs in the same way as DataStream
> >> jobs, or else users need to write their own main class.
> >
> > Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> > DataStream API also doesn't provide a default main class for users,
> > why do we need to provide such one for SQL?
> >
> >> With the help of ExecNodeGraph, do we still need the serialized
> >> SessionState? If not, we could make SQL Driver accepts two serialized
> >> formats:
> >
> > No, ExecNodeGraph doesn't need to serialize SessionState. I thought the
> > proposed SqlDriver was a dedicated main class accepting SQL files, is
> > that correct?
> > If true, we have to ship the SessionState for this case which is a large
> > work.
> > I think we just need a JsonPlanDriver which is a main class that accepts
> > JsonPlan as the parameter.
> >
> >
> >> The common solutions I know is to use distributed file systems or use
> >> init containers to localize the resources.
> >
> > Personally, I prefer the way of init containers which doesn't depend on
> > additional components.
> > This can reduce the moving parts of a production environment.
> > Depending on a distributed file system makes the testing, demo, and local
> > setup harder than init containers.
> >
> > Best,
> > Jark
> >
> >
> >
> >
> > On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com>> wrote:
> >
> >> The FLIP is in the early phase and some details are not included, but
> >> fortunately, we got lots of valuable ideas from the discussion.
> >>
> >> Thanks to everyone who joined the dissuasion!
> >> @Weihua @Shanmon @Shengkai @Biao @Jark
> >>
> >> This weekend I’m gonna revisit and update the FLIP, adding more
> >> details. Hopefully, we can further align our opinions.
> >>
> >> Best,
> >> Paul Lam
> >>
> >>> 2023年6月2日 18:02，Paul Lam <pa...@gmail.com> 写道：
> >>>
> >>> Hi Jark,
> >>>
> >>> Thanks a lot for your input!
> >>>
> >>>> If we decide to submit ExecNodeGraph instead of SQL file, is it still
> >>>> necessary to support SQL Driver?
> >>>
> >>> I think so. Apart from usage in SQL Gateway, SQL Driver could simplify
> >>> Flink SQL execution with Flink CLI.
> >>>
> >>> As the FLIP said, it’s good to have a default main class for Flink
> SQLs,
> >>> which allows users to submit Flink SQLs in the same way as DataStream
> >>> jobs, or else users need to write their own main class.
> >>>
> >>>> SQL Driver needs to serialize SessionState which is very challenging
> >>>> but not detailed covered in the FLIP.
> >>>
> >>> With the help of ExecNodeGraph, do we still need the serialized
> >>> SessionState? If not, we could make SQL Driver accepts two serialized
> >>> formats:
> >>>
> >>> - SQL files for user-facing public usage
> >>> - ExecNodeGraph for internal usage
> >>>
> >>> It’s kind of similar to the relationship between job jars and
> jobgraphs.
> >>>
> >>>> Regarding "K8S doesn't support shipping multiple jars", is that true?
> >> Is it
> >>>> possible to support it?
> >>>
> >>> Yes, K8s doesn’t distribute any files. It’s the users’ responsibility
> to
> >> make
> >>> sure the resources are accessible in the containers. The common
> solutions
> >>> I know is to use distributed file systems or use init containers to
> >> localize the
> >>> resources.
> >>>
> >>> Now I lean toward introducing a fs to do the distribution job. WDYT?
> >>>
> >>> Best,
> >>> Paul Lam
> >>>
> >>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com>
> <mailto:imjark@gmail.com <ma...@gmail.com>>>
> >> 写道：
> >>>>
> >>>> Hi Paul,
> >>>>
> >>>> Thanks for starting this discussion. I like the proposal! This is a
> >>>> frequently requested feature!
> >>>>
> >>>> I agree with Shengkai that ExecNodeGraph as the submission object is a
> >>>> better idea than SQL file. To be more specific, it should be
> >> JsonPlanGraph
> >>>> or CompiledPlan which is the serializable representation. CompiledPlan
> >> is a
> >>>> clear separation between compiling/optimization/validation and
> >> execution.
> >>>> This can keep the validation and metadata accessing still on the
> >> SQLGateway
> >>>> side. This allows SQLGateway to leverage some metadata caching and UDF
> >> JAR
> >>>> caching for better compiling performance.
> >>>>
> >>>> If we decide to submit ExecNodeGraph instead of SQL file, is it still
> >>>> necessary to support SQL Driver? Regarding non-interactive SQL jobs,
> >> users
> >>>> can use the Table API program for application mode. SQL Driver needs
> to
> >>>> serialize SessionState which is very challenging but not detailed
> >> covered
> >>>> in the FLIP.
> >>>>
> >>>> Regarding "K8S doesn't support shipping multiple jars", is that true?
> >> Is it
> >>>> possible to support it?
> >>>>
> >>>> Best,
> >>>> Jark
> >>>>
> >>>>
> >>>>
> >>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com> <mailto:
> >> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
> >>>>
> >>>>> Hi Weihua,
> >>>>>
> >>>>> You’re right. Distributing the SQLs to the TMs is one of the
> >> challenging
> >>>>> parts of this FLIP.
> >>>>>
> >>>>> Web submission is not enabled in application mode currently as you
> >> said,
> >>>>> but it could be changed if we have good reasons.
> >>>>>
> >>>>> What do you think about introducing a distributed storage for SQL
> >> Gateway?
> >>>>>
> >>>>> We could make use of Flink file systems [1] to distribute the SQL
> >> Gateway
> >>>>> generated resources, that should solve the problem at its root cause.
> >>>>>
> >>>>> Users could specify Flink-supported file systems to ship files. It’s
> >> only
> >>>>> required when using SQL Gateway with K8s application mode.
> >>>>>
> >>>>> [1]
> >>>>>
> >>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >> <
> >>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >>>
> >>>>>
> >>>>> Best,
> >>>>> Paul Lam
> >>>>>
> >>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> huweihua.ckl@gmail.com>> 写道：
> >>>>>>
> >>>>>> Thanks Paul for your reply.
> >>>>>>
> >>>>>> SQLDriver looks good to me.
> >>>>>>
> >>>>>> 2. Do you mean a pass the SQL string a configuration or a program
> >>>>> argument?
> >>>>>>
> >>>>>>
> >>>>>> I brought this up because we were unable to pass the SQL file to
> Flink
> >>>>>> using Kubernetes mode.
> >>>>>> For DataStream/Python users, they need to prepare their images for
> the
> >>>>> jars
> >>>>>> and dependencies.
> >>>>>> But for SQL users, they can use a common image to run different SQL
> >>>>> queries
> >>>>>> if there are no other udf requirements.
> >>>>>> It would be great if the SQL query and image were not bound.
> >>>>>>
> >>>>>> Using strings is a way to decouple these, but just as you mentioned,
> >> it's
> >>>>>> not easy to pass complex SQL.
> >>>>>>
> >>>>>>> use web submission
> >>>>>> AFAIK, we can not use web submission in the Application mode. Please
> >>>>>> correct me if I'm wrong.
> >>>>>>
> >>>>>>
> >>>>>> Best,
> >>>>>> Weihua
> >>>>>>
> >>>>>>
> >>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com>>
> >> wrote:
> >>>>>>
> >>>>>>> Hi Biao,
> >>>>>>>
> >>>>>>> Thanks for your comments!
> >>>>>>>
> >>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL
> >> jobs
> >>>>>>> in
> >>>>>>>> Application mode? More specifically, if we use SQL client/gateway
> to
> >>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
> flink
> >> to
> >>>>>>> use
> >>>>>>>> Application mode to execute those queries after this FLIP?
> >>>>>>>
> >>>>>>> Thanks for pointing it out. I think only DMLs would be executed via
> >> SQL
> >>>>>>> Driver.
> >>>>>>> I'll add the scope to the FLIP.
> >>>>>>>
> >>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> >> trivial as
> >>>>>>> we
> >>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
> >> more
> >>>>>>>> complicated as Shengkai said.
> >>>>>>>
> >>>>>>>
> >>>>>>> Your input is very informative. I’m thinking about using web
> >> submission,
> >>>>>>> but it requires exposing the JobManager port which could also be a
> >>>>> problem
> >>>>>>> on K8s.
> >>>>>>>
> >>>>>>> Another approach is to explicitly require a distributed storage to
> >> ship
> >>>>>>> files,
> >>>>>>> but we may need a new deployment executor for that.
> >>>>>>>
> >>>>>>> What do you think of these two approaches?
> >>>>>>>
> >>>>>>>> 3. Serialization of SessionState: in SessionState, there are some
> >>>>>>>> unserializable fields
> >>>>>>>> like
> >> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >>>>> It
> >>>>>>>> may be worthwhile to add more details about the serialization
> part.
> >>>>>>>
> >>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as
> >> Shengkai
> >>>>>>> mentioned, do we eliminate the need for serialization of
> >> SessionState?
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <mailto:
> biaogeng7@gmail.com>> 写道：
> >>>>>>>>
> >>>>>>>> Thanks Paul for the proposal!I believe it would be very useful for
> >>>>> flink
> >>>>>>>> users.
> >>>>>>>> After reading the FLIP, I have some questions:
> >>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL
> >> jobs
> >>>>>>> in
> >>>>>>>> Application mode? More specifically, if we use SQL client/gateway
> to
> >>>>>>>> execute some interactive SQLs like a SELECT query, can we ask
> flink
> >> to
> >>>>>>> use
> >>>>>>>> Application mode to execute those queries after this FLIP?
> >>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> >> trivial as
> >>>>>>> we
> >>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
> >> more
> >>>>>>>> complicated as Shengkai said. I have implemented a simple POC
> >>>>>>>> <
> >>>>>>>
> >>>>>
> >>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> <
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> >>>>>>>>
> >>>>>>>> based on SQL client before(i.e. consider the SQL client which
> >> supports
> >>>>>>>> executing a SQL file as the SQL driver in this FLIP). One problem
> I
> >>>>> have
> >>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
> >>>>> Without
> >>>>>>>> such support, users have to modify the initContainer or rebuild a
> >> new
> >>>>> K8s
> >>>>>>>> image every time to fetch the SQL file. Like the flink k8s
> operator,
> >>>>> one
> >>>>>>>> workaround is to utilize the flink config(transforming the SQL
> file
> >> to
> >>>>> a
> >>>>>>>> escaped string like Weihua mentioned) which will be converted to a
> >>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
> >>>>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/ <
> https://kubernetes.io/docs/concepts/configuration/configmap/>>).
> >> Not
> >>>>>>> sure
> >>>>>>>> if we have better solutions.
> >>>>>>>> 3. Serialization of SessionState: in SessionState, there are some
> >>>>>>>> unserializable fields
> >>>>>>>> like
> >> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >>>>> It
> >>>>>>>> may be worthwhile to add more details about the serialization
> part.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Biao Geng
> >>>>>>>>
> >>>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>>
> 于2023年5月31日周三 11:49写道：
> >>>>>>>>
> >>>>>>>>> Hi Weihua,
> >>>>>>>>>
> >>>>>>>>> Thanks a lot for your input! Please see my comments inline.
> >>>>>>>>>
> >>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job.
> (Not
> >>>>>>>>> strong,
> >>>>>>>>>> the SQLDriver is fine for me)
> >>>>>>>>>
> >>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
> >> following
> >>>>>>>>> reasons FYI:
> >>>>>>>>>
> >>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
> >>>>>>>>> 2. Flink program's main class is sort of like Driver in JDBC
> which
> >>>>>>>>> translates SQLs into
> >>>>>>>>> databases specific languages.
> >>>>>>>>>
> >>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >>>>>>>>>
> >>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need
> >> to
> >>>>>>>>> prepare
> >>>>>>>>>> a SQL file in an image for Kubernetes application mode, which
> may
> >> be
> >>>>> a
> >>>>>>>>> bit
> >>>>>>>>>> cumbersome.
> >>>>>>>>>
> >>>>>>>>> Do you mean a pass the SQL string a configuration or a program
> >>>>> argument?
> >>>>>>>>>
> >>>>>>>>> I thought it might be convenient for testing propose, but not
> >>>>>>> recommended
> >>>>>>>>> for production,
> >>>>>>>>> cause Flink SQLs could be complicated and involves lots of
> >> characters
> >>>>>>> that
> >>>>>>>>> need to escape.
> >>>>>>>>>
> >>>>>>>>> WDYT?
> >>>>>>>>>
> >>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>>>>>>> "run-application"
> >>>>>>>>>> command. Does that mean we need to perform automatic detection
> in
> >>>>>>> Flink?
> >>>>>>>>>
> >>>>>>>>> Yes! It’s like running a PyFlink job with the following command:
> >>>>>>>>>
> >>>>>>>>> ```
> >>>>>>>>> ./bin/flink run \
> >>>>>>>>> --pyModule table.word_count \
> >>>>>>>>> --pyFiles examples/python/table
> >>>>>>>>> ```
> >>>>>>>>>
> >>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
> >>>>>>>>> automatically.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> <
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Paul Lam
> >>>>>>>>>
> >>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <mailto:
> huweihua.ckl@gmail.com>> 写道：
> >>>>>>>>>>
> >>>>>>>>>> Thanks Paul for the proposal.
> >>>>>>>>>>
> >>>>>>>>>> +1 for this. It is valuable in improving ease of use.
> >>>>>>>>>>
> >>>>>>>>>> I have a few questions.
> >>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job.
> (Not
> >>>>>>>>> strong,
> >>>>>>>>>> the SQLDriver is fine for me)
> >>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need
> >> to
> >>>>>>>>> prepare
> >>>>>>>>>> a SQL file in an image for Kubernetes application mode, which
> may
> >> be
> >>>>> a
> >>>>>>>>> bit
> >>>>>>>>>> cumbersome.
> >>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>>>>>>> "run-application"
> >>>>>>>>>> command. Does that mean we need to perform automatic detection
> in
> >>>>>>> Flink?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Weihua
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <paullin3280@gmail.com
> <ma...@gmail.com>>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi team,
> >>>>>>>>>>>
> >>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
> >> introduces
> >>>>> a
> >>>>>>>>> SQL
> >>>>>>>>>>> driver as the
> >>>>>>>>>>> default main class for Flink SQL jobs.
> >>>>>>>>>>>
> >>>>>>>>>>> Currently, Flink SQL could be executed out of the box either
> via
> >> SQL
> >>>>>>>>>>> Client/Gateway
> >>>>>>>>>>> or embedded in a Flink Java/Python program.
> >>>>>>>>>>>
> >>>>>>>>>>> However, each one has its drawback:
> >>>>>>>>>>>
> >>>>>>>>>>> - SQL Client/Gateway doesn’t support the application deployment
> >> mode
> >>>>>>> [2]
> >>>>>>>>>>> - Flink Java/Python program requires extra work to write a
> >> non-SQL
> >>>>>>>>> program
> >>>>>>>>>>>
> >>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the default
> >> main
> >>>>>>>>> class
> >>>>>>>>>>> for SQL jobs.
> >>>>>>>>>>> Please see the FLIP docs for details and feel free to comment.
> >>>>> Thanks!
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>>>>>>>>>>
> >>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> >>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Paul Lam
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Jark,
Thanks for your input! Please see my comments inline.
> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> DataStream API also doesn't provide a default main class for users,
> why do we need to provide such one for SQL?
Sorry for the confusion I caused. By DataStream jobs, I mean jobs submitted
via Flink CLI which actually could be DataStream/Table jobs.
I think a default main class would be user-friendly which eliminates the need
for users to write a main class as SQLRunner in Flink K8s operator [1].
> I thought the proposed SqlDriver was a dedicated main class accepting SQL files, is
> that correct?
Both JSON plans and SQL files are accepted. SQL Gateway should use JSON plans,
while CLI users may use either JSON plans or SQL files.
Please see the updated FLIP[2] for more details.
> Personally, I prefer the way of init containers which doesn't depend on
> additional components.
> This can reduce the moving parts of a production environment.
> Depending on a distributed file system makes the testing, demo, and local
> setup harder than init containers.
Please note that we could reuse the checkpoint storage like S3/HDFS, which should
be required to run Flink in production, so I guess that would be acceptable for most
users. WDYT?
WRT testing, demo, and local setups, I think we could support the local filesystem
scheme i.e. file://** as the state backends do. It works as long as SQL Gateway
and JobManager(or SQL Driver) can access the resource directory (specified via
`sql-gateway.application.storage-dir`).
Thanks!
[1] https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
[3] https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161
Best,
Paul Lam
> 2023年6月3日 12:21，Jark Wu <im...@gmail.com> 写道：
>
> Hi Paul,
>
> Thanks for your reply. I left my comments inline.
>
>> As the FLIP said, it’s good to have a default main class for Flink SQLs,
>> which allows users to submit Flink SQLs in the same way as DataStream
>> jobs, or else users need to write their own main class.
>
> Isn't Table API the same way as DataSream jobs to submit Flink SQL?
> DataStream API also doesn't provide a default main class for users,
> why do we need to provide such one for SQL?
>
>> With the help of ExecNodeGraph, do we still need the serialized
>> SessionState? If not, we could make SQL Driver accepts two serialized
>> formats:
>
> No, ExecNodeGraph doesn't need to serialize SessionState. I thought the
> proposed SqlDriver was a dedicated main class accepting SQL files, is
> that correct?
> If true, we have to ship the SessionState for this case which is a large
> work.
> I think we just need a JsonPlanDriver which is a main class that accepts
> JsonPlan as the parameter.
>
>
>> The common solutions I know is to use distributed file systems or use
>> init containers to localize the resources.
>
> Personally, I prefer the way of init containers which doesn't depend on
> additional components.
> This can reduce the moving parts of a production environment.
> Depending on a distributed file system makes the testing, demo, and local
> setup harder than init containers.
>
> Best,
> Jark
>
>
>
>
> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
>
>> The FLIP is in the early phase and some details are not included, but
>> fortunately, we got lots of valuable ideas from the discussion.
>>
>> Thanks to everyone who joined the dissuasion!
>> @Weihua @Shanmon @Shengkai @Biao @Jark
>>
>> This weekend I’m gonna revisit and update the FLIP, adding more
>> details. Hopefully, we can further align our opinions.
>>
>> Best,
>> Paul Lam
>>
>>> 2023年6月2日 18:02，Paul Lam <pa...@gmail.com> 写道：
>>>
>>> Hi Jark,
>>>
>>> Thanks a lot for your input!
>>>
>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it still
>>>> necessary to support SQL Driver?
>>>
>>> I think so. Apart from usage in SQL Gateway, SQL Driver could simplify
>>> Flink SQL execution with Flink CLI.
>>>
>>> As the FLIP said, it’s good to have a default main class for Flink SQLs,
>>> which allows users to submit Flink SQLs in the same way as DataStream
>>> jobs, or else users need to write their own main class.
>>>
>>>> SQL Driver needs to serialize SessionState which is very challenging
>>>> but not detailed covered in the FLIP.
>>>
>>> With the help of ExecNodeGraph, do we still need the serialized
>>> SessionState? If not, we could make SQL Driver accepts two serialized
>>> formats:
>>>
>>> - SQL files for user-facing public usage
>>> - ExecNodeGraph for internal usage
>>>
>>> It’s kind of similar to the relationship between job jars and jobgraphs.
>>>
>>>> Regarding "K8S doesn't support shipping multiple jars", is that true?
>> Is it
>>>> possible to support it?
>>>
>>> Yes, K8s doesn’t distribute any files. It’s the users’ responsibility to
>> make
>>> sure the resources are accessible in the containers. The common solutions
>>> I know is to use distributed file systems or use init containers to
>> localize the
>>> resources.
>>>
>>> Now I lean toward introducing a fs to do the distribution job. WDYT?
>>>
>>> Best,
>>> Paul Lam
>>>
>>>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com> <mailto:imjark@gmail.com <ma...@gmail.com>>>
>> 写道：
>>>>
>>>> Hi Paul,
>>>>
>>>> Thanks for starting this discussion. I like the proposal! This is a
>>>> frequently requested feature!
>>>>
>>>> I agree with Shengkai that ExecNodeGraph as the submission object is a
>>>> better idea than SQL file. To be more specific, it should be
>> JsonPlanGraph
>>>> or CompiledPlan which is the serializable representation. CompiledPlan
>> is a
>>>> clear separation between compiling/optimization/validation and
>> execution.
>>>> This can keep the validation and metadata accessing still on the
>> SQLGateway
>>>> side. This allows SQLGateway to leverage some metadata caching and UDF
>> JAR
>>>> caching for better compiling performance.
>>>>
>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it still
>>>> necessary to support SQL Driver? Regarding non-interactive SQL jobs,
>> users
>>>> can use the Table API program for application mode. SQL Driver needs to
>>>> serialize SessionState which is very challenging but not detailed
>> covered
>>>> in the FLIP.
>>>>
>>>> Regarding "K8S doesn't support shipping multiple jars", is that true?
>> Is it
>>>> possible to support it?
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>>
>>>>
>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <ma...@gmail.com> <mailto:
>> paullin3280@gmail.com <ma...@gmail.com>>> wrote:
>>>>
>>>>> Hi Weihua,
>>>>>
>>>>> You’re right. Distributing the SQLs to the TMs is one of the
>> challenging
>>>>> parts of this FLIP.
>>>>>
>>>>> Web submission is not enabled in application mode currently as you
>> said,
>>>>> but it could be changed if we have good reasons.
>>>>>
>>>>> What do you think about introducing a distributed storage for SQL
>> Gateway?
>>>>>
>>>>> We could make use of Flink file systems [1] to distribute the SQL
>> Gateway
>>>>> generated resources, that should solve the problem at its root cause.
>>>>>
>>>>> Users could specify Flink-supported file systems to ship files. It’s
>> only
>>>>> required when using SQL Gateway with K8s application mode.
>>>>>
>>>>> [1]
>>>>>
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>> <
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com>> 写道：
>>>>>>
>>>>>> Thanks Paul for your reply.
>>>>>>
>>>>>> SQLDriver looks good to me.
>>>>>>
>>>>>> 2. Do you mean a pass the SQL string a configuration or a program
>>>>> argument?
>>>>>>
>>>>>>
>>>>>> I brought this up because we were unable to pass the SQL file to Flink
>>>>>> using Kubernetes mode.
>>>>>> For DataStream/Python users, they need to prepare their images for the
>>>>> jars
>>>>>> and dependencies.
>>>>>> But for SQL users, they can use a common image to run different SQL
>>>>> queries
>>>>>> if there are no other udf requirements.
>>>>>> It would be great if the SQL query and image were not bound.
>>>>>>
>>>>>> Using strings is a way to decouple these, but just as you mentioned,
>> it's
>>>>>> not easy to pass complex SQL.
>>>>>>
>>>>>>> use web submission
>>>>>> AFAIK, we can not use web submission in the Application mode. Please
>>>>>> correct me if I'm wrong.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Weihua
>>>>>>
>>>>>>
>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>>
>> wrote:
>>>>>>
>>>>>>> Hi Biao,
>>>>>>>
>>>>>>> Thanks for your comments!
>>>>>>>
>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL
>> jobs
>>>>>>> in
>>>>>>>> Application mode? More specifically, if we use SQL client/gateway to
>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask flink
>> to
>>>>>>> use
>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>
>>>>>>> Thanks for pointing it out. I think only DMLs would be executed via
>> SQL
>>>>>>> Driver.
>>>>>>> I'll add the scope to the FLIP.
>>>>>>>
>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>> trivial as
>>>>>>> we
>>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
>> more
>>>>>>>> complicated as Shengkai said.
>>>>>>>
>>>>>>>
>>>>>>> Your input is very informative. I’m thinking about using web
>> submission,
>>>>>>> but it requires exposing the JobManager port which could also be a
>>>>> problem
>>>>>>> on K8s.
>>>>>>>
>>>>>>> Another approach is to explicitly require a distributed storage to
>> ship
>>>>>>> files,
>>>>>>> but we may need a new deployment executor for that.
>>>>>>>
>>>>>>> What do you think of these two approaches?
>>>>>>>
>>>>>>>> 3. Serialization of SessionState: in SessionState, there are some
>>>>>>>> unserializable fields
>>>>>>>> like
>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>> It
>>>>>>>> may be worthwhile to add more details about the serialization part.
>>>>>>>
>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as
>> Shengkai
>>>>>>> mentioned, do we eliminate the need for serialization of
>> SessionState?
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaogeng7@gmail.com <ma...@gmail.com>> 写道：
>>>>>>>>
>>>>>>>> Thanks Paul for the proposal!I believe it would be very useful for
>>>>> flink
>>>>>>>> users.
>>>>>>>> After reading the FLIP, I have some questions:
>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL
>> jobs
>>>>>>> in
>>>>>>>> Application mode? More specifically, if we use SQL client/gateway to
>>>>>>>> execute some interactive SQLs like a SELECT query, can we ask flink
>> to
>>>>>>> use
>>>>>>>> Application mode to execute those queries after this FLIP?
>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is
>> trivial as
>>>>>>> we
>>>>>>>> can ship files via YARN's tool easily but for K8s, things can be
>> more
>>>>>>>> complicated as Shengkai said. I have implemented a simple POC
>>>>>>>> <
>>>>>>>
>>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
>>>>>>>>
>>>>>>>> based on SQL client before(i.e. consider the SQL client which
>> supports
>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One problem I
>>>>> have
>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
>>>>> Without
>>>>>>>> such support, users have to modify the initContainer or rebuild a
>> new
>>>>> K8s
>>>>>>>> image every time to fetch the SQL file. Like the flink k8s operator,
>>>>> one
>>>>>>>> workaround is to utilize the flink config(transforming the SQL file
>> to
>>>>> a
>>>>>>>> escaped string like Weihua mentioned) which will be converted to a
>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/ <https://kubernetes.io/docs/concepts/configuration/configmap/>>).
>> Not
>>>>>>> sure
>>>>>>>> if we have better solutions.
>>>>>>>> 3. Serialization of SessionState: in SessionState, there are some
>>>>>>>> unserializable fields
>>>>>>>> like
>> org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>>>> It
>>>>>>>> may be worthwhile to add more details about the serialization part.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Biao Geng
>>>>>>>>
>>>>>>>> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> 于2023年5月31日周三 11:49写道：
>>>>>>>>
>>>>>>>>> Hi Weihua,
>>>>>>>>>
>>>>>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>>>>>
>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>>>>>>>>> strong,
>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>
>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
>> following
>>>>>>>>> reasons FYI:
>>>>>>>>>
>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>>>>>> 2. Flink program's main class is sort of like Driver in JDBC which
>>>>>>>>> translates SQLs into
>>>>>>>>> databases specific languages.
>>>>>>>>>
>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>>>>>
>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need
>> to
>>>>>>>>> prepare
>>>>>>>>>> a SQL file in an image for Kubernetes application mode, which may
>> be
>>>>> a
>>>>>>>>> bit
>>>>>>>>>> cumbersome.
>>>>>>>>>
>>>>>>>>> Do you mean a pass the SQL string a configuration or a program
>>>>> argument?
>>>>>>>>>
>>>>>>>>> I thought it might be convenient for testing propose, but not
>>>>>>> recommended
>>>>>>>>> for production,
>>>>>>>>> cause Flink SQLs could be complicated and involves lots of
>> characters
>>>>>>> that
>>>>>>>>> need to escape.
>>>>>>>>>
>>>>>>>>> WDYT?
>>>>>>>>>
>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>> "run-application"
>>>>>>>>>> command. Does that mean we need to perform automatic detection in
>>>>>>> Flink?
>>>>>>>>>
>>>>>>>>> Yes! It’s like running a PyFlink job with the following command:
>>>>>>>>>
>>>>>>>>> ```
>>>>>>>>> ./bin/flink run \
>>>>>>>>> --pyModule table.word_count \
>>>>>>>>> --pyFiles examples/python/table
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
>>>>>>>>> automatically.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>>
>>>>>>>
>>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java <https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Paul Lam
>>>>>>>>>
>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua.ckl@gmail.com <ma...@gmail.com>> 写道：
>>>>>>>>>>
>>>>>>>>>> Thanks Paul for the proposal.
>>>>>>>>>>
>>>>>>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>>>>>>
>>>>>>>>>> I have a few questions.
>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>>>>>>>>> strong,
>>>>>>>>>> the SQLDriver is fine for me)
>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need
>> to
>>>>>>>>> prepare
>>>>>>>>>> a SQL file in an image for Kubernetes application mode, which may
>> be
>>>>> a
>>>>>>>>> bit
>>>>>>>>>> cumbersome.
>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>>>> "run-application"
>>>>>>>>>> command. Does that mean we need to perform automatic detection in
>>>>>>> Flink?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Weihua
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi team,
>>>>>>>>>>>
>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
>> introduces
>>>>> a
>>>>>>>>> SQL
>>>>>>>>>>> driver as the
>>>>>>>>>>> default main class for Flink SQL jobs.
>>>>>>>>>>>
>>>>>>>>>>> Currently, Flink SQL could be executed out of the box either via
>> SQL
>>>>>>>>>>> Client/Gateway
>>>>>>>>>>> or embedded in a Flink Java/Python program.
>>>>>>>>>>>
>>>>>>>>>>> However, each one has its drawback:
>>>>>>>>>>>
>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application deployment
>> mode
>>>>>>> [2]
>>>>>>>>>>> - Flink Java/Python program requires extra work to write a
>> non-SQL
>>>>>>>>> program
>>>>>>>>>>>
>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the default
>> main
>>>>>>>>> class
>>>>>>>>>>> for SQL jobs.
>>>>>>>>>>> Please see the FLIP docs for details and feel free to comment.
>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>>>>>>>>
>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Paul Lam
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Jark Wu <im...@gmail.com>.
Hi Paul,
Thanks for your reply. I left my comments inline.
> As the FLIP said, it’s good to have a default main class for Flink SQLs,
> which allows users to submit Flink SQLs in the same way as DataStream
> jobs, or else users need to write their own main class.
Isn't Table API the same way as DataSream jobs to submit Flink SQL?
DataStream API also doesn't provide a default main class for users,
why do we need to provide such one for SQL?
> With the help of ExecNodeGraph, do we still need the serialized
> SessionState? If not, we could make SQL Driver accepts two serialized
> formats:
No, ExecNodeGraph doesn't need to serialize SessionState. I thought the
proposed SqlDriver was a dedicated main class accepting SQL files, is
that correct?
If true, we have to ship the SessionState for this case which is a large
work.
I think we just need a JsonPlanDriver which is a main class that accepts
JsonPlan as the parameter.
> The common solutions I know is to use distributed file systems or use
> init containers to localize the resources.
Personally, I prefer the way of init containers which doesn't depend on
additional components.
This can reduce the moving parts of a production environment.
Depending on a distributed file system makes the testing, demo, and local
setup harder than init containers.
Best,
Jark
On Fri, 2 Jun 2023 at 18:10, Paul Lam <pa...@gmail.com> wrote:
> The FLIP is in the early phase and some details are not included, but
> fortunately, we got lots of valuable ideas from the discussion.
>
> Thanks to everyone who joined the dissuasion!
> @Weihua @Shanmon @Shengkai @Biao @Jark
>
> This weekend I’m gonna revisit and update the FLIP, adding more
> details. Hopefully, we can further align our opinions.
>
> Best,
> Paul Lam
>
> > 2023年6月2日 18:02，Paul Lam <pa...@gmail.com> 写道：
> >
> > Hi Jark,
> >
> > Thanks a lot for your input!
> >
> >> If we decide to submit ExecNodeGraph instead of SQL file, is it still
> >> necessary to support SQL Driver?
> >
> > I think so. Apart from usage in SQL Gateway, SQL Driver could simplify
> > Flink SQL execution with Flink CLI.
> >
> > As the FLIP said, it’s good to have a default main class for Flink SQLs,
> > which allows users to submit Flink SQLs in the same way as DataStream
> > jobs, or else users need to write their own main class.
> >
> >> SQL Driver needs to serialize SessionState which is very challenging
> >> but not detailed covered in the FLIP.
> >
> > With the help of ExecNodeGraph, do we still need the serialized
> > SessionState? If not, we could make SQL Driver accepts two serialized
> > formats:
> >
> > - SQL files for user-facing public usage
> > - ExecNodeGraph for internal usage
> >
> > It’s kind of similar to the relationship between job jars and jobgraphs.
> >
> >> Regarding "K8S doesn't support shipping multiple jars", is that true?
> Is it
> >> possible to support it?
> >
> > Yes, K8s doesn’t distribute any files. It’s the users’ responsibility to
> make
> > sure the resources are accessible in the containers. The common solutions
> > I know is to use distributed file systems or use init containers to
> localize the
> > resources.
> >
> > Now I lean toward introducing a fs to do the distribution job. WDYT?
> >
> > Best,
> > Paul Lam
> >
> >> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com>>
> 写道：
> >>
> >> Hi Paul,
> >>
> >> Thanks for starting this discussion. I like the proposal! This is a
> >> frequently requested feature!
> >>
> >> I agree with Shengkai that ExecNodeGraph as the submission object is a
> >> better idea than SQL file. To be more specific, it should be
> JsonPlanGraph
> >> or CompiledPlan which is the serializable representation. CompiledPlan
> is a
> >> clear separation between compiling/optimization/validation and
> execution.
> >> This can keep the validation and metadata accessing still on the
> SQLGateway
> >> side. This allows SQLGateway to leverage some metadata caching and UDF
> JAR
> >> caching for better compiling performance.
> >>
> >> If we decide to submit ExecNodeGraph instead of SQL file, is it still
> >> necessary to support SQL Driver? Regarding non-interactive SQL jobs,
> users
> >> can use the Table API program for application mode. SQL Driver needs to
> >> serialize SessionState which is very challenging but not detailed
> covered
> >> in the FLIP.
> >>
> >> Regarding "K8S doesn't support shipping multiple jars", is that true?
> Is it
> >> possible to support it?
> >>
> >> Best,
> >> Jark
> >>
> >>
> >>
> >> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <mailto:
> paullin3280@gmail.com>> wrote:
> >>
> >>> Hi Weihua,
> >>>
> >>> You’re right. Distributing the SQLs to the TMs is one of the
> challenging
> >>> parts of this FLIP.
> >>>
> >>> Web submission is not enabled in application mode currently as you
> said,
> >>> but it could be changed if we have good reasons.
> >>>
> >>> What do you think about introducing a distributed storage for SQL
> Gateway?
> >>>
> >>> We could make use of Flink file systems [1] to distribute the SQL
> Gateway
> >>> generated resources, that should solve the problem at its root cause.
> >>>
> >>> Users could specify Flink-supported file systems to ship files. It’s
> only
> >>> required when using SQL Gateway with K8s application mode.
> >>>
> >>> [1]
> >>>
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> <
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
> >
> >>>
> >>> Best,
> >>> Paul Lam
> >>>
> >>>> 2023年6月1日 13:55，Weihua Hu <hu...@gmail.com> 写道：
> >>>>
> >>>> Thanks Paul for your reply.
> >>>>
> >>>> SQLDriver looks good to me.
> >>>>
> >>>> 2. Do you mean a pass the SQL string a configuration or a program
> >>> argument?
> >>>>
> >>>>
> >>>> I brought this up because we were unable to pass the SQL file to Flink
> >>>> using Kubernetes mode.
> >>>> For DataStream/Python users, they need to prepare their images for the
> >>> jars
> >>>> and dependencies.
> >>>> But for SQL users, they can use a common image to run different SQL
> >>> queries
> >>>> if there are no other udf requirements.
> >>>> It would be great if the SQL query and image were not bound.
> >>>>
> >>>> Using strings is a way to decouple these, but just as you mentioned,
> it's
> >>>> not easy to pass complex SQL.
> >>>>
> >>>>> use web submission
> >>>> AFAIK, we can not use web submission in the Application mode. Please
> >>>> correct me if I'm wrong.
> >>>>
> >>>>
> >>>> Best,
> >>>> Weihua
> >>>>
> >>>>
> >>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <pa...@gmail.com>
> wrote:
> >>>>
> >>>>> Hi Biao,
> >>>>>
> >>>>> Thanks for your comments!
> >>>>>
> >>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL
> jobs
> >>>>> in
> >>>>>> Application mode? More specifically, if we use SQL client/gateway to
> >>>>>> execute some interactive SQLs like a SELECT query, can we ask flink
> to
> >>>>> use
> >>>>>> Application mode to execute those queries after this FLIP?
> >>>>>
> >>>>> Thanks for pointing it out. I think only DMLs would be executed via
> SQL
> >>>>> Driver.
> >>>>> I'll add the scope to the FLIP.
> >>>>>
> >>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> trivial as
> >>>>> we
> >>>>>> can ship files via YARN's tool easily but for K8s, things can be
> more
> >>>>>> complicated as Shengkai said.
> >>>>>
> >>>>>
> >>>>> Your input is very informative. I’m thinking about using web
> submission,
> >>>>> but it requires exposing the JobManager port which could also be a
> >>> problem
> >>>>> on K8s.
> >>>>>
> >>>>> Another approach is to explicitly require a distributed storage to
> ship
> >>>>> files,
> >>>>> but we may need a new deployment executor for that.
> >>>>>
> >>>>> What do you think of these two approaches?
> >>>>>
> >>>>>> 3. Serialization of SessionState: in SessionState, there are some
> >>>>>> unserializable fields
> >>>>>> like
> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >>> It
> >>>>>> may be worthwhile to add more details about the serialization part.
> >>>>>
> >>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as
> Shengkai
> >>>>> mentioned, do we eliminate the need for serialization of
> SessionState?
> >>>>>
> >>>>> Best,
> >>>>> Paul Lam
> >>>>>
> >>>>>> 2023年5月31日 13:07，Biao Geng <bi...@gmail.com> 写道：
> >>>>>>
> >>>>>> Thanks Paul for the proposal!I believe it would be very useful for
> >>> flink
> >>>>>> users.
> >>>>>> After reading the FLIP, I have some questions:
> >>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL
> jobs
> >>>>> in
> >>>>>> Application mode? More specifically, if we use SQL client/gateway to
> >>>>>> execute some interactive SQLs like a SELECT query, can we ask flink
> to
> >>>>> use
> >>>>>> Application mode to execute those queries after this FLIP?
> >>>>>> 2. Deployment: I believe in YARN mode, the implementation is
> trivial as
> >>>>> we
> >>>>>> can ship files via YARN's tool easily but for K8s, things can be
> more
> >>>>>> complicated as Shengkai said. I have implemented a simple POC
> >>>>>> <
> >>>>>
> >>>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>>>>
> >>>>>> based on SQL client before(i.e. consider the SQL client which
> supports
> >>>>>> executing a SQL file as the SQL driver in this FLIP). One problem I
> >>> have
> >>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
> >>> Without
> >>>>>> such support, users have to modify the initContainer or rebuild a
> new
> >>> K8s
> >>>>>> image every time to fetch the SQL file. Like the flink k8s operator,
> >>> one
> >>>>>> workaround is to utilize the flink config(transforming the SQL file
> to
> >>> a
> >>>>>> escaped string like Weihua mentioned) which will be converted to a
> >>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
> >>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/>).
> Not
> >>>>> sure
> >>>>>> if we have better solutions.
> >>>>>> 3. Serialization of SessionState: in SessionState, there are some
> >>>>>> unserializable fields
> >>>>>> like
> org.apache.flink.table.resource.ResourceManager#userClassLoader.
> >>> It
> >>>>>> may be worthwhile to add more details about the serialization part.
> >>>>>>
> >>>>>> Best,
> >>>>>> Biao Geng
> >>>>>>
> >>>>>> Paul Lam <pa...@gmail.com> 于2023年5月31日周三 11:49写道：
> >>>>>>
> >>>>>>> Hi Weihua,
> >>>>>>>
> >>>>>>> Thanks a lot for your input! Please see my comments inline.
> >>>>>>>
> >>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> >>>>>>> strong,
> >>>>>>>> the SQLDriver is fine for me)
> >>>>>>>
> >>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the
> following
> >>>>>>> reasons FYI:
> >>>>>>>
> >>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
> >>>>>>> 2. Flink program's main class is sort of like Driver in JDBC which
> >>>>>>> translates SQLs into
> >>>>>>> databases specific languages.
> >>>>>>>
> >>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >>>>>>>
> >>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need
> to
> >>>>>>> prepare
> >>>>>>>> a SQL file in an image for Kubernetes application mode, which may
> be
> >>> a
> >>>>>>> bit
> >>>>>>>> cumbersome.
> >>>>>>>
> >>>>>>> Do you mean a pass the SQL string a configuration or a program
> >>> argument?
> >>>>>>>
> >>>>>>> I thought it might be convenient for testing propose, but not
> >>>>> recommended
> >>>>>>> for production,
> >>>>>>> cause Flink SQLs could be complicated and involves lots of
> characters
> >>>>> that
> >>>>>>> need to escape.
> >>>>>>>
> >>>>>>> WDYT?
> >>>>>>>
> >>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>>>>> "run-application"
> >>>>>>>> command. Does that mean we need to perform automatic detection in
> >>>>> Flink?
> >>>>>>>
> >>>>>>> Yes! It’s like running a PyFlink job with the following command:
> >>>>>>>
> >>>>>>> ```
> >>>>>>> ./bin/flink run \
> >>>>>>> --pyModule table.word_count \
> >>>>>>> --pyFiles examples/python/table
> >>>>>>> ```
> >>>>>>>
> >>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
> >>>>>>> automatically.
> >>>>>>>
> >>>>>>>
> >>>>>>> [1]
> >>>>>>>
> >>>>>
> >>>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Paul Lam
> >>>>>>>
> >>>>>>>> 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
> >>>>>>>>
> >>>>>>>> Thanks Paul for the proposal.
> >>>>>>>>
> >>>>>>>> +1 for this. It is valuable in improving ease of use.
> >>>>>>>>
> >>>>>>>> I have a few questions.
> >>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> >>>>>>> strong,
> >>>>>>>> the SQLDriver is fine for me)
> >>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need
> to
> >>>>>>> prepare
> >>>>>>>> a SQL file in an image for Kubernetes application mode, which may
> be
> >>> a
> >>>>>>> bit
> >>>>>>>> cumbersome.
> >>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>>>>> "run-application"
> >>>>>>>> command. Does that mean we need to perform automatic detection in
> >>>>> Flink?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Weihua
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi team,
> >>>>>>>>>
> >>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which
> introduces
> >>> a
> >>>>>>> SQL
> >>>>>>>>> driver as the
> >>>>>>>>> default main class for Flink SQL jobs.
> >>>>>>>>>
> >>>>>>>>> Currently, Flink SQL could be executed out of the box either via
> SQL
> >>>>>>>>> Client/Gateway
> >>>>>>>>> or embedded in a Flink Java/Python program.
> >>>>>>>>>
> >>>>>>>>> However, each one has its drawback:
> >>>>>>>>>
> >>>>>>>>> - SQL Client/Gateway doesn’t support the application deployment
> mode
> >>>>> [2]
> >>>>>>>>> - Flink Java/Python program requires extra work to write a
> non-SQL
> >>>>>>> program
> >>>>>>>>>
> >>>>>>>>> Therefore, I propose adding a SQL driver to act as the default
> main
> >>>>>>> class
> >>>>>>>>> for SQL jobs.
> >>>>>>>>> Please see the FLIP docs for details and feel free to comment.
> >>> Thanks!
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >>>>>>>>> <
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>>>>>>>>
> >>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> >>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Paul Lam
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
The FLIP is in the early phase and some details are not included, but
fortunately, we got lots of valuable ideas from the discussion.
Thanks to everyone who joined the dissuasion!
@Weihua @Shanmon @Shengkai @Biao @Jark
This weekend I’m gonna revisit and update the FLIP, adding more
details. Hopefully, we can further align our opinions.
Best,
Paul Lam
> 2023年6月2日 18:02，Paul Lam <pa...@gmail.com> 写道：
>
> Hi Jark,
>
> Thanks a lot for your input!
>
>> If we decide to submit ExecNodeGraph instead of SQL file, is it still
>> necessary to support SQL Driver?
>
> I think so. Apart from usage in SQL Gateway, SQL Driver could simplify
> Flink SQL execution with Flink CLI.
>
> As the FLIP said, it’s good to have a default main class for Flink SQLs,
> which allows users to submit Flink SQLs in the same way as DataStream
> jobs, or else users need to write their own main class.
>
>> SQL Driver needs to serialize SessionState which is very challenging
>> but not detailed covered in the FLIP.
>
> With the help of ExecNodeGraph, do we still need the serialized
> SessionState? If not, we could make SQL Driver accepts two serialized
> formats:
>
> - SQL files for user-facing public usage
> - ExecNodeGraph for internal usage
>
> It’s kind of similar to the relationship between job jars and jobgraphs.
>
>> Regarding "K8S doesn't support shipping multiple jars", is that true? Is it
>> possible to support it?
>
> Yes, K8s doesn’t distribute any files. It’s the users’ responsibility to make
> sure the resources are accessible in the containers. The common solutions
> I know is to use distributed file systems or use init containers to localize the
> resources.
>
> Now I lean toward introducing a fs to do the distribution job. WDYT?
>
> Best,
> Paul Lam
>
>> 2023年6月1日 20:33，Jark Wu <imjark@gmail.com <ma...@gmail.com>> 写道：
>>
>> Hi Paul,
>>
>> Thanks for starting this discussion. I like the proposal! This is a
>> frequently requested feature!
>>
>> I agree with Shengkai that ExecNodeGraph as the submission object is a
>> better idea than SQL file. To be more specific, it should be JsonPlanGraph
>> or CompiledPlan which is the serializable representation. CompiledPlan is a
>> clear separation between compiling/optimization/validation and execution.
>> This can keep the validation and metadata accessing still on the SQLGateway
>> side. This allows SQLGateway to leverage some metadata caching and UDF JAR
>> caching for better compiling performance.
>>
>> If we decide to submit ExecNodeGraph instead of SQL file, is it still
>> necessary to support SQL Driver? Regarding non-interactive SQL jobs, users
>> can use the Table API program for application mode. SQL Driver needs to
>> serialize SessionState which is very challenging but not detailed covered
>> in the FLIP.
>>
>> Regarding "K8S doesn't support shipping multiple jars", is that true? Is it
>> possible to support it?
>>
>> Best,
>> Jark
>>
>>
>>
>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
>>
>>> Hi Weihua,
>>>
>>> You’re right. Distributing the SQLs to the TMs is one of the challenging
>>> parts of this FLIP.
>>>
>>> Web submission is not enabled in application mode currently as you said,
>>> but it could be changed if we have good reasons.
>>>
>>> What do you think about introducing a distributed storage for SQL Gateway?
>>>
>>> We could make use of Flink file systems [1] to distribute the SQL Gateway
>>> generated resources, that should solve the problem at its root cause.
>>>
>>> Users could specify Flink-supported file systems to ship files. It’s only
>>> required when using SQL Gateway with K8s application mode.
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ <https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/>
>>>
>>> Best,
>>> Paul Lam
>>>
>>>> 2023年6月1日 13:55，Weihua Hu <hu...@gmail.com> 写道：
>>>>
>>>> Thanks Paul for your reply.
>>>>
>>>> SQLDriver looks good to me.
>>>>
>>>> 2. Do you mean a pass the SQL string a configuration or a program
>>> argument?
>>>>
>>>>
>>>> I brought this up because we were unable to pass the SQL file to Flink
>>>> using Kubernetes mode.
>>>> For DataStream/Python users, they need to prepare their images for the
>>> jars
>>>> and dependencies.
>>>> But for SQL users, they can use a common image to run different SQL
>>> queries
>>>> if there are no other udf requirements.
>>>> It would be great if the SQL query and image were not bound.
>>>>
>>>> Using strings is a way to decouple these, but just as you mentioned, it's
>>>> not easy to pass complex SQL.
>>>>
>>>>> use web submission
>>>> AFAIK, we can not use web submission in the Application mode. Please
>>>> correct me if I'm wrong.
>>>>
>>>>
>>>> Best,
>>>> Weihua
>>>>
>>>>
>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <pa...@gmail.com> wrote:
>>>>
>>>>> Hi Biao,
>>>>>
>>>>> Thanks for your comments!
>>>>>
>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
>>>>> in
>>>>>> Application mode? More specifically, if we use SQL client/gateway to
>>>>>> execute some interactive SQLs like a SELECT query, can we ask flink to
>>>>> use
>>>>>> Application mode to execute those queries after this FLIP?
>>>>>
>>>>> Thanks for pointing it out. I think only DMLs would be executed via SQL
>>>>> Driver.
>>>>> I'll add the scope to the FLIP.
>>>>>
>>>>>> 2. Deployment: I believe in YARN mode, the implementation is trivial as
>>>>> we
>>>>>> can ship files via YARN's tool easily but for K8s, things can be more
>>>>>> complicated as Shengkai said.
>>>>>
>>>>>
>>>>> Your input is very informative. I’m thinking about using web submission,
>>>>> but it requires exposing the JobManager port which could also be a
>>> problem
>>>>> on K8s.
>>>>>
>>>>> Another approach is to explicitly require a distributed storage to ship
>>>>> files,
>>>>> but we may need a new deployment executor for that.
>>>>>
>>>>> What do you think of these two approaches?
>>>>>
>>>>>> 3. Serialization of SessionState: in SessionState, there are some
>>>>>> unserializable fields
>>>>>> like org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>> It
>>>>>> may be worthwhile to add more details about the serialization part.
>>>>>
>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as Shengkai
>>>>> mentioned, do we eliminate the need for serialization of SessionState?
>>>>>
>>>>> Best,
>>>>> Paul Lam
>>>>>
>>>>>> 2023年5月31日 13:07，Biao Geng <bi...@gmail.com> 写道：
>>>>>>
>>>>>> Thanks Paul for the proposal!I believe it would be very useful for
>>> flink
>>>>>> users.
>>>>>> After reading the FLIP, I have some questions:
>>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
>>>>> in
>>>>>> Application mode? More specifically, if we use SQL client/gateway to
>>>>>> execute some interactive SQLs like a SELECT query, can we ask flink to
>>>>> use
>>>>>> Application mode to execute those queries after this FLIP?
>>>>>> 2. Deployment: I believe in YARN mode, the implementation is trivial as
>>>>> we
>>>>>> can ship files via YARN's tool easily but for K8s, things can be more
>>>>>> complicated as Shengkai said. I have implemented a simple POC
>>>>>> <
>>>>>
>>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>>>
>>>>>> based on SQL client before(i.e. consider the SQL client which supports
>>>>>> executing a SQL file as the SQL driver in this FLIP). One problem I
>>> have
>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
>>> Without
>>>>>> such support, users have to modify the initContainer or rebuild a new
>>> K8s
>>>>>> image every time to fetch the SQL file. Like the flink k8s operator,
>>> one
>>>>>> workaround is to utilize the flink config(transforming the SQL file to
>>> a
>>>>>> escaped string like Weihua mentioned) which will be converted to a
>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
>>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/>). Not
>>>>> sure
>>>>>> if we have better solutions.
>>>>>> 3. Serialization of SessionState: in SessionState, there are some
>>>>>> unserializable fields
>>>>>> like org.apache.flink.table.resource.ResourceManager#userClassLoader.
>>> It
>>>>>> may be worthwhile to add more details about the serialization part.
>>>>>>
>>>>>> Best,
>>>>>> Biao Geng
>>>>>>
>>>>>> Paul Lam <pa...@gmail.com> 于2023年5月31日周三 11:49写道：
>>>>>>
>>>>>>> Hi Weihua,
>>>>>>>
>>>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>>>
>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>>>>>>> strong,
>>>>>>>> the SQLDriver is fine for me)
>>>>>>>
>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the following
>>>>>>> reasons FYI:
>>>>>>>
>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>>>> 2. Flink program's main class is sort of like Driver in JDBC which
>>>>>>> translates SQLs into
>>>>>>> databases specific languages.
>>>>>>>
>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>>>
>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>>>>>>> prepare
>>>>>>>> a SQL file in an image for Kubernetes application mode, which may be
>>> a
>>>>>>> bit
>>>>>>>> cumbersome.
>>>>>>>
>>>>>>> Do you mean a pass the SQL string a configuration or a program
>>> argument?
>>>>>>>
>>>>>>> I thought it might be convenient for testing propose, but not
>>>>> recommended
>>>>>>> for production,
>>>>>>> cause Flink SQLs could be complicated and involves lots of characters
>>>>> that
>>>>>>> need to escape.
>>>>>>>
>>>>>>> WDYT?
>>>>>>>
>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>> "run-application"
>>>>>>>> command. Does that mean we need to perform automatic detection in
>>>>> Flink?
>>>>>>>
>>>>>>> Yes! It’s like running a PyFlink job with the following command:
>>>>>>>
>>>>>>> ```
>>>>>>> ./bin/flink run \
>>>>>>> --pyModule table.word_count \
>>>>>>> --pyFiles examples/python/table
>>>>>>> ```
>>>>>>>
>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
>>>>>>> automatically.
>>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>>
>>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>>>>
>>>>>>> Best,
>>>>>>> Paul Lam
>>>>>>>
>>>>>>>> 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
>>>>>>>>
>>>>>>>> Thanks Paul for the proposal.
>>>>>>>>
>>>>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>>>>
>>>>>>>> I have a few questions.
>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>>>>>>> strong,
>>>>>>>> the SQLDriver is fine for me)
>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>>>>>>> prepare
>>>>>>>> a SQL file in an image for Kubernetes application mode, which may be
>>> a
>>>>>>> bit
>>>>>>>> cumbersome.
>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>>> "run-application"
>>>>>>>> command. Does that mean we need to perform automatic detection in
>>>>> Flink?
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Weihua
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi team,
>>>>>>>>>
>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which introduces
>>> a
>>>>>>> SQL
>>>>>>>>> driver as the
>>>>>>>>> default main class for Flink SQL jobs.
>>>>>>>>>
>>>>>>>>> Currently, Flink SQL could be executed out of the box either via SQL
>>>>>>>>> Client/Gateway
>>>>>>>>> or embedded in a Flink Java/Python program.
>>>>>>>>>
>>>>>>>>> However, each one has its drawback:
>>>>>>>>>
>>>>>>>>> - SQL Client/Gateway doesn’t support the application deployment mode
>>>>> [2]
>>>>>>>>> - Flink Java/Python program requires extra work to write a non-SQL
>>>>>>> program
>>>>>>>>>
>>>>>>>>> Therefore, I propose adding a SQL driver to act as the default main
>>>>>>> class
>>>>>>>>> for SQL jobs.
>>>>>>>>> Please see the FLIP docs for details and feel free to comment.
>>> Thanks!
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>>
>>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>>>>>>
>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Paul Lam
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Jark,
Thanks a lot for your input!
> If we decide to submit ExecNodeGraph instead of SQL file, is it still
> necessary to support SQL Driver?
I think so. Apart from usage in SQL Gateway, SQL Driver could simplify
Flink SQL execution with Flink CLI.
As the FLIP said, it’s good to have a default main class for Flink SQLs,
which allows users to submit Flink SQLs in the same way as DataStream
jobs, or else users need to write their own main class.
> SQL Driver needs to serialize SessionState which is very challenging
> but not detailed covered in the FLIP.
With the help of ExecNodeGraph, do we still need the serialized
SessionState? If not, we could make SQL Driver accepts two serialized
formats:
- SQL files for user-facing public usage
- ExecNodeGraph for internal usage
It’s kind of similar to the relationship between job jars and jobgraphs.
> Regarding "K8S doesn't support shipping multiple jars", is that true? Is it
> possible to support it?
Yes, K8s doesn’t distribute any files. It’s the users’ responsibility to make
sure the resources are accessible in the containers. The common solutions
I know is to use distributed file systems or use init containers to localize the
resources.
Now I lean toward introducing a fs to do the distribution job. WDYT?
Best,
Paul Lam
> 2023年6月1日 20:33，Jark Wu <im...@gmail.com> 写道：
>
> Hi Paul,
>
> Thanks for starting this discussion. I like the proposal! This is a
> frequently requested feature!
>
> I agree with Shengkai that ExecNodeGraph as the submission object is a
> better idea than SQL file. To be more specific, it should be JsonPlanGraph
> or CompiledPlan which is the serializable representation. CompiledPlan is a
> clear separation between compiling/optimization/validation and execution.
> This can keep the validation and metadata accessing still on the SQLGateway
> side. This allows SQLGateway to leverage some metadata caching and UDF JAR
> caching for better compiling performance.
>
> If we decide to submit ExecNodeGraph instead of SQL file, is it still
> necessary to support SQL Driver? Regarding non-interactive SQL jobs, users
> can use the Table API program for application mode. SQL Driver needs to
> serialize SessionState which is very challenging but not detailed covered
> in the FLIP.
>
> Regarding "K8S doesn't support shipping multiple jars", is that true? Is it
> possible to support it?
>
> Best,
> Jark
>
>
>
> On Thu, 1 Jun 2023 at 16:58, Paul Lam <pa...@gmail.com> wrote:
>
>> Hi Weihua,
>>
>> You’re right. Distributing the SQLs to the TMs is one of the challenging
>> parts of this FLIP.
>>
>> Web submission is not enabled in application mode currently as you said,
>> but it could be changed if we have good reasons.
>>
>> What do you think about introducing a distributed storage for SQL Gateway?
>>
>> We could make use of Flink file systems [1] to distribute the SQL Gateway
>> generated resources, that should solve the problem at its root cause.
>>
>> Users could specify Flink-supported file systems to ship files. It’s only
>> required when using SQL Gateway with K8s application mode.
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>>
>> Best,
>> Paul Lam
>>
>>> 2023年6月1日 13:55，Weihua Hu <hu...@gmail.com> 写道：
>>>
>>> Thanks Paul for your reply.
>>>
>>> SQLDriver looks good to me.
>>>
>>> 2. Do you mean a pass the SQL string a configuration or a program
>> argument?
>>>
>>>
>>> I brought this up because we were unable to pass the SQL file to Flink
>>> using Kubernetes mode.
>>> For DataStream/Python users, they need to prepare their images for the
>> jars
>>> and dependencies.
>>> But for SQL users, they can use a common image to run different SQL
>> queries
>>> if there are no other udf requirements.
>>> It would be great if the SQL query and image were not bound.
>>>
>>> Using strings is a way to decouple these, but just as you mentioned, it's
>>> not easy to pass complex SQL.
>>>
>>>> use web submission
>>> AFAIK, we can not use web submission in the Application mode. Please
>>> correct me if I'm wrong.
>>>
>>>
>>> Best,
>>> Weihua
>>>
>>>
>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam <pa...@gmail.com> wrote:
>>>
>>>> Hi Biao,
>>>>
>>>> Thanks for your comments!
>>>>
>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
>>>> in
>>>>> Application mode? More specifically, if we use SQL client/gateway to
>>>>> execute some interactive SQLs like a SELECT query, can we ask flink to
>>>> use
>>>>> Application mode to execute those queries after this FLIP?
>>>>
>>>> Thanks for pointing it out. I think only DMLs would be executed via SQL
>>>> Driver.
>>>> I'll add the scope to the FLIP.
>>>>
>>>>> 2. Deployment: I believe in YARN mode, the implementation is trivial as
>>>> we
>>>>> can ship files via YARN's tool easily but for K8s, things can be more
>>>>> complicated as Shengkai said.
>>>>
>>>>
>>>> Your input is very informative. I’m thinking about using web submission,
>>>> but it requires exposing the JobManager port which could also be a
>> problem
>>>> on K8s.
>>>>
>>>> Another approach is to explicitly require a distributed storage to ship
>>>> files,
>>>> but we may need a new deployment executor for that.
>>>>
>>>> What do you think of these two approaches?
>>>>
>>>>> 3. Serialization of SessionState: in SessionState, there are some
>>>>> unserializable fields
>>>>> like org.apache.flink.table.resource.ResourceManager#userClassLoader.
>> It
>>>>> may be worthwhile to add more details about the serialization part.
>>>>
>>>> I agree. That’s a missing part. But if we use ExecNodeGraph as Shengkai
>>>> mentioned, do we eliminate the need for serialization of SessionState?
>>>>
>>>> Best,
>>>> Paul Lam
>>>>
>>>>> 2023年5月31日 13:07，Biao Geng <bi...@gmail.com> 写道：
>>>>>
>>>>> Thanks Paul for the proposal!I believe it would be very useful for
>> flink
>>>>> users.
>>>>> After reading the FLIP, I have some questions:
>>>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
>>>> in
>>>>> Application mode? More specifically, if we use SQL client/gateway to
>>>>> execute some interactive SQLs like a SELECT query, can we ask flink to
>>>> use
>>>>> Application mode to execute those queries after this FLIP?
>>>>> 2. Deployment: I believe in YARN mode, the implementation is trivial as
>>>> we
>>>>> can ship files via YARN's tool easily but for K8s, things can be more
>>>>> complicated as Shengkai said. I have implemented a simple POC
>>>>> <
>>>>
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>>>
>>>>> based on SQL client before(i.e. consider the SQL client which supports
>>>>> executing a SQL file as the SQL driver in this FLIP). One problem I
>> have
>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
>> Without
>>>>> such support, users have to modify the initContainer or rebuild a new
>> K8s
>>>>> image every time to fetch the SQL file. Like the flink k8s operator,
>> one
>>>>> workaround is to utilize the flink config(transforming the SQL file to
>> a
>>>>> escaped string like Weihua mentioned) which will be converted to a
>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
>>>>> <https://kubernetes.io/docs/concepts/configuration/configmap/>). Not
>>>> sure
>>>>> if we have better solutions.
>>>>> 3. Serialization of SessionState: in SessionState, there are some
>>>>> unserializable fields
>>>>> like org.apache.flink.table.resource.ResourceManager#userClassLoader.
>> It
>>>>> may be worthwhile to add more details about the serialization part.
>>>>>
>>>>> Best,
>>>>> Biao Geng
>>>>>
>>>>> Paul Lam <pa...@gmail.com> 于2023年5月31日周三 11:49写道：
>>>>>
>>>>>> Hi Weihua,
>>>>>>
>>>>>> Thanks a lot for your input! Please see my comments inline.
>>>>>>
>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>>>>>> strong,
>>>>>>> the SQLDriver is fine for me)
>>>>>>
>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the following
>>>>>> reasons FYI:
>>>>>>
>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>>>> 2. Flink program's main class is sort of like Driver in JDBC which
>>>>>> translates SQLs into
>>>>>> databases specific languages.
>>>>>>
>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>>>
>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>>>>>> prepare
>>>>>>> a SQL file in an image for Kubernetes application mode, which may be
>> a
>>>>>> bit
>>>>>>> cumbersome.
>>>>>>
>>>>>> Do you mean a pass the SQL string a configuration or a program
>> argument?
>>>>>>
>>>>>> I thought it might be convenient for testing propose, but not
>>>> recommended
>>>>>> for production,
>>>>>> cause Flink SQLs could be complicated and involves lots of characters
>>>> that
>>>>>> need to escape.
>>>>>>
>>>>>> WDYT?
>>>>>>
>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>> "run-application"
>>>>>>> command. Does that mean we need to perform automatic detection in
>>>> Flink?
>>>>>>
>>>>>> Yes! It’s like running a PyFlink job with the following command:
>>>>>>
>>>>>> ```
>>>>>> ./bin/flink run \
>>>>>> --pyModule table.word_count \
>>>>>> --pyFiles examples/python/table
>>>>>> ```
>>>>>>
>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
>>>>>> automatically.
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>>
>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>>>
>>>>>> Best,
>>>>>> Paul Lam
>>>>>>
>>>>>>> 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
>>>>>>>
>>>>>>> Thanks Paul for the proposal.
>>>>>>>
>>>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>>>
>>>>>>> I have a few questions.
>>>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>>>>>> strong,
>>>>>>> the SQLDriver is fine for me)
>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>>>>>> prepare
>>>>>>> a SQL file in an image for Kubernetes application mode, which may be
>> a
>>>>>> bit
>>>>>>> cumbersome.
>>>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>>>> "run-application"
>>>>>>> command. Does that mean we need to perform automatic detection in
>>>> Flink?
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Weihua
>>>>>>>
>>>>>>>
>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>>> Hi team,
>>>>>>>>
>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which introduces
>> a
>>>>>> SQL
>>>>>>>> driver as the
>>>>>>>> default main class for Flink SQL jobs.
>>>>>>>>
>>>>>>>> Currently, Flink SQL could be executed out of the box either via SQL
>>>>>>>> Client/Gateway
>>>>>>>> or embedded in a Flink Java/Python program.
>>>>>>>>
>>>>>>>> However, each one has its drawback:
>>>>>>>>
>>>>>>>> - SQL Client/Gateway doesn’t support the application deployment mode
>>>> [2]
>>>>>>>> - Flink Java/Python program requires extra work to write a non-SQL
>>>>>> program
>>>>>>>>
>>>>>>>> Therefore, I propose adding a SQL driver to act as the default main
>>>>>> class
>>>>>>>> for SQL jobs.
>>>>>>>> Please see the FLIP docs for details and feel free to comment.
>> Thanks!
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>>>>> <
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>>>>>
>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Paul Lam
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Jark Wu <im...@gmail.com>.
Hi Paul,
Thanks for starting this discussion. I like the proposal! This is a
frequently requested feature!
I agree with Shengkai that ExecNodeGraph as the submission object is a
better idea than SQL file. To be more specific, it should be JsonPlanGraph
or CompiledPlan which is the serializable representation. CompiledPlan is a
clear separation between compiling/optimization/validation and execution.
This can keep the validation and metadata accessing still on the SQLGateway
side. This allows SQLGateway to leverage some metadata caching and UDF JAR
caching for better compiling performance.
If we decide to submit ExecNodeGraph instead of SQL file, is it still
necessary to support SQL Driver? Regarding non-interactive SQL jobs, users
can use the Table API program for application mode. SQL Driver needs to
serialize SessionState which is very challenging but not detailed covered
in the FLIP.
Regarding "K8S doesn't support shipping multiple jars", is that true? Is it
possible to support it?
Best,
Jark
On Thu, 1 Jun 2023 at 16:58, Paul Lam <pa...@gmail.com> wrote:
> Hi Weihua,
>
> You’re right. Distributing the SQLs to the TMs is one of the challenging
> parts of this FLIP.
>
> Web submission is not enabled in application mode currently as you said,
> but it could be changed if we have good reasons.
>
> What do you think about introducing a distributed storage for SQL Gateway?
>
> We could make use of Flink file systems [1] to distribute the SQL Gateway
> generated resources, that should solve the problem at its root cause.
>
> Users could specify Flink-supported file systems to ship files. It’s only
> required when using SQL Gateway with K8s application mode.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
>
> Best,
> Paul Lam
>
> > 2023年6月1日 13:55，Weihua Hu <hu...@gmail.com> 写道：
> >
> > Thanks Paul for your reply.
> >
> > SQLDriver looks good to me.
> >
> > 2. Do you mean a pass the SQL string a configuration or a program
> argument?
> >
> >
> > I brought this up because we were unable to pass the SQL file to Flink
> > using Kubernetes mode.
> > For DataStream/Python users, they need to prepare their images for the
> jars
> > and dependencies.
> > But for SQL users, they can use a common image to run different SQL
> queries
> > if there are no other udf requirements.
> > It would be great if the SQL query and image were not bound.
> >
> > Using strings is a way to decouple these, but just as you mentioned, it's
> > not easy to pass complex SQL.
> >
> >> use web submission
> > AFAIK, we can not use web submission in the Application mode. Please
> > correct me if I'm wrong.
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Wed, May 31, 2023 at 9:37 PM Paul Lam <pa...@gmail.com> wrote:
> >
> >> Hi Biao,
> >>
> >> Thanks for your comments!
> >>
> >>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
> >> in
> >>> Application mode? More specifically, if we use SQL client/gateway to
> >>> execute some interactive SQLs like a SELECT query, can we ask flink to
> >> use
> >>> Application mode to execute those queries after this FLIP?
> >>
> >> Thanks for pointing it out. I think only DMLs would be executed via SQL
> >> Driver.
> >> I'll add the scope to the FLIP.
> >>
> >>> 2. Deployment: I believe in YARN mode, the implementation is trivial as
> >> we
> >>> can ship files via YARN's tool easily but for K8s, things can be more
> >>> complicated as Shengkai said.
> >>
> >>
> >> Your input is very informative. I’m thinking about using web submission,
> >> but it requires exposing the JobManager port which could also be a
> problem
> >> on K8s.
> >>
> >> Another approach is to explicitly require a distributed storage to ship
> >> files,
> >> but we may need a new deployment executor for that.
> >>
> >> What do you think of these two approaches?
> >>
> >>> 3. Serialization of SessionState: in SessionState, there are some
> >>> unserializable fields
> >>> like org.apache.flink.table.resource.ResourceManager#userClassLoader.
> It
> >>> may be worthwhile to add more details about the serialization part.
> >>
> >> I agree. That’s a missing part. But if we use ExecNodeGraph as Shengkai
> >> mentioned, do we eliminate the need for serialization of SessionState?
> >>
> >> Best,
> >> Paul Lam
> >>
> >>> 2023年5月31日 13:07，Biao Geng <bi...@gmail.com> 写道：
> >>>
> >>> Thanks Paul for the proposal!I believe it would be very useful for
> flink
> >>> users.
> >>> After reading the FLIP, I have some questions:
> >>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
> >> in
> >>> Application mode? More specifically, if we use SQL client/gateway to
> >>> execute some interactive SQLs like a SELECT query, can we ask flink to
> >> use
> >>> Application mode to execute those queries after this FLIP?
> >>> 2. Deployment: I believe in YARN mode, the implementation is trivial as
> >> we
> >>> can ship files via YARN's tool easily but for K8s, things can be more
> >>> complicated as Shengkai said. I have implemented a simple POC
> >>> <
> >>
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >>>
> >>> based on SQL client before(i.e. consider the SQL client which supports
> >>> executing a SQL file as the SQL driver in this FLIP). One problem I
> have
> >>> met is how do we ship SQL files ( or Job Graph) to the k8s side.
> Without
> >>> such support, users have to modify the initContainer or rebuild a new
> K8s
> >>> image every time to fetch the SQL file. Like the flink k8s operator,
> one
> >>> workaround is to utilize the flink config(transforming the SQL file to
> a
> >>> escaped string like Weihua mentioned) which will be converted to a
> >>> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
> >>> <https://kubernetes.io/docs/concepts/configuration/configmap/>). Not
> >> sure
> >>> if we have better solutions.
> >>> 3. Serialization of SessionState: in SessionState, there are some
> >>> unserializable fields
> >>> like org.apache.flink.table.resource.ResourceManager#userClassLoader.
> It
> >>> may be worthwhile to add more details about the serialization part.
> >>>
> >>> Best,
> >>> Biao Geng
> >>>
> >>> Paul Lam <pa...@gmail.com> 于2023年5月31日周三 11:49写道：
> >>>
> >>>> Hi Weihua,
> >>>>
> >>>> Thanks a lot for your input! Please see my comments inline.
> >>>>
> >>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> >>>> strong,
> >>>>> the SQLDriver is fine for me)
> >>>>
> >>>> I’ve thought about SQL Runner but picked SQL Driver for the following
> >>>> reasons FYI:
> >>>>
> >>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
> >>>> 2. Flink program's main class is sort of like Driver in JDBC which
> >>>> translates SQLs into
> >>>> databases specific languages.
> >>>>
> >>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >>>>
> >>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
> >>>> prepare
> >>>>> a SQL file in an image for Kubernetes application mode, which may be
> a
> >>>> bit
> >>>>> cumbersome.
> >>>>
> >>>> Do you mean a pass the SQL string a configuration or a program
> argument?
> >>>>
> >>>> I thought it might be convenient for testing propose, but not
> >> recommended
> >>>> for production,
> >>>> cause Flink SQLs could be complicated and involves lots of characters
> >> that
> >>>> need to escape.
> >>>>
> >>>> WDYT?
> >>>>
> >>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>> "run-application"
> >>>>> command. Does that mean we need to perform automatic detection in
> >> Flink?
> >>>>
> >>>> Yes! It’s like running a PyFlink job with the following command:
> >>>>
> >>>> ```
> >>>> ./bin/flink run \
> >>>> --pyModule table.word_count \
> >>>> --pyFiles examples/python/table
> >>>> ```
> >>>>
> >>>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
> >>>> automatically.
> >>>>
> >>>>
> >>>> [1]
> >>>>
> >>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>>>
> >>>> Best,
> >>>> Paul Lam
> >>>>
> >>>>> 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
> >>>>>
> >>>>> Thanks Paul for the proposal.
> >>>>>
> >>>>> +1 for this. It is valuable in improving ease of use.
> >>>>>
> >>>>> I have a few questions.
> >>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> >>>> strong,
> >>>>> the SQLDriver is fine for me)
> >>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
> >>>> prepare
> >>>>> a SQL file in an image for Kubernetes application mode, which may be
> a
> >>>> bit
> >>>>> cumbersome.
> >>>>> - I noticed that we don't specify the SQLDriver jar in the
> >>>> "run-application"
> >>>>> command. Does that mean we need to perform automatic detection in
> >> Flink?
> >>>>>
> >>>>>
> >>>>> Best,
> >>>>> Weihua
> >>>>>
> >>>>>
> >>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> Hi team,
> >>>>>>
> >>>>>> I’d like to start a discussion about FLIP-316 [1], which introduces
> a
> >>>> SQL
> >>>>>> driver as the
> >>>>>> default main class for Flink SQL jobs.
> >>>>>>
> >>>>>> Currently, Flink SQL could be executed out of the box either via SQL
> >>>>>> Client/Gateway
> >>>>>> or embedded in a Flink Java/Python program.
> >>>>>>
> >>>>>> However, each one has its drawback:
> >>>>>>
> >>>>>> - SQL Client/Gateway doesn’t support the application deployment mode
> >> [2]
> >>>>>> - Flink Java/Python program requires extra work to write a non-SQL
> >>>> program
> >>>>>>
> >>>>>> Therefore, I propose adding a SQL driver to act as the default main
> >>>> class
> >>>>>> for SQL jobs.
> >>>>>> Please see the FLIP docs for details and feel free to comment.
> Thanks!
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >>>>>> <
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>>>>>
> >>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> >>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
> >>>>>>
> >>>>>> Best,
> >>>>>> Paul Lam
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Weihua,
You’re right. Distributing the SQLs to the TMs is one of the challenging
parts of this FLIP.
Web submission is not enabled in application mode currently as you said,
but it could be changed if we have good reasons.
What do you think about introducing a distributed storage for SQL Gateway?
We could make use of Flink file systems [1] to distribute the SQL Gateway
generated resources, that should solve the problem at its root cause.
Users could specify Flink-supported file systems to ship files. It’s only
required when using SQL Gateway with K8s application mode.
[1] https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/
Best,
Paul Lam
> 2023年6月1日 13:55，Weihua Hu <hu...@gmail.com> 写道：
>
> Thanks Paul for your reply.
>
> SQLDriver looks good to me.
>
> 2. Do you mean a pass the SQL string a configuration or a program argument?
>
>
> I brought this up because we were unable to pass the SQL file to Flink
> using Kubernetes mode.
> For DataStream/Python users, they need to prepare their images for the jars
> and dependencies.
> But for SQL users, they can use a common image to run different SQL queries
> if there are no other udf requirements.
> It would be great if the SQL query and image were not bound.
>
> Using strings is a way to decouple these, but just as you mentioned, it's
> not easy to pass complex SQL.
>
>> use web submission
> AFAIK, we can not use web submission in the Application mode. Please
> correct me if I'm wrong.
>
>
> Best,
> Weihua
>
>
> On Wed, May 31, 2023 at 9:37 PM Paul Lam <pa...@gmail.com> wrote:
>
>> Hi Biao,
>>
>> Thanks for your comments!
>>
>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
>> in
>>> Application mode? More specifically, if we use SQL client/gateway to
>>> execute some interactive SQLs like a SELECT query, can we ask flink to
>> use
>>> Application mode to execute those queries after this FLIP?
>>
>> Thanks for pointing it out. I think only DMLs would be executed via SQL
>> Driver.
>> I'll add the scope to the FLIP.
>>
>>> 2. Deployment: I believe in YARN mode, the implementation is trivial as
>> we
>>> can ship files via YARN's tool easily but for K8s, things can be more
>>> complicated as Shengkai said.
>>
>>
>> Your input is very informative. I’m thinking about using web submission,
>> but it requires exposing the JobManager port which could also be a problem
>> on K8s.
>>
>> Another approach is to explicitly require a distributed storage to ship
>> files,
>> but we may need a new deployment executor for that.
>>
>> What do you think of these two approaches?
>>
>>> 3. Serialization of SessionState: in SessionState, there are some
>>> unserializable fields
>>> like org.apache.flink.table.resource.ResourceManager#userClassLoader. It
>>> may be worthwhile to add more details about the serialization part.
>>
>> I agree. That’s a missing part. But if we use ExecNodeGraph as Shengkai
>> mentioned, do we eliminate the need for serialization of SessionState?
>>
>> Best,
>> Paul Lam
>>
>>> 2023年5月31日 13:07，Biao Geng <bi...@gmail.com> 写道：
>>>
>>> Thanks Paul for the proposal!I believe it would be very useful for flink
>>> users.
>>> After reading the FLIP, I have some questions:
>>> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
>> in
>>> Application mode? More specifically, if we use SQL client/gateway to
>>> execute some interactive SQLs like a SELECT query, can we ask flink to
>> use
>>> Application mode to execute those queries after this FLIP?
>>> 2. Deployment: I believe in YARN mode, the implementation is trivial as
>> we
>>> can ship files via YARN's tool easily but for K8s, things can be more
>>> complicated as Shengkai said. I have implemented a simple POC
>>> <
>> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
>>>
>>> based on SQL client before(i.e. consider the SQL client which supports
>>> executing a SQL file as the SQL driver in this FLIP). One problem I have
>>> met is how do we ship SQL files ( or Job Graph) to the k8s side. Without
>>> such support, users have to modify the initContainer or rebuild a new K8s
>>> image every time to fetch the SQL file. Like the flink k8s operator, one
>>> workaround is to utilize the flink config(transforming the SQL file to a
>>> escaped string like Weihua mentioned) which will be converted to a
>>> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
>>> <https://kubernetes.io/docs/concepts/configuration/configmap/>). Not
>> sure
>>> if we have better solutions.
>>> 3. Serialization of SessionState: in SessionState, there are some
>>> unserializable fields
>>> like org.apache.flink.table.resource.ResourceManager#userClassLoader. It
>>> may be worthwhile to add more details about the serialization part.
>>>
>>> Best,
>>> Biao Geng
>>>
>>> Paul Lam <pa...@gmail.com> 于2023年5月31日周三 11:49写道：
>>>
>>>> Hi Weihua,
>>>>
>>>> Thanks a lot for your input! Please see my comments inline.
>>>>
>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>>>> strong,
>>>>> the SQLDriver is fine for me)
>>>>
>>>> I’ve thought about SQL Runner but picked SQL Driver for the following
>>>> reasons FYI:
>>>>
>>>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>>>> 2. Flink program's main class is sort of like Driver in JDBC which
>>>> translates SQLs into
>>>> databases specific languages.
>>>>
>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>>>
>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>>>> prepare
>>>>> a SQL file in an image for Kubernetes application mode, which may be a
>>>> bit
>>>>> cumbersome.
>>>>
>>>> Do you mean a pass the SQL string a configuration or a program argument?
>>>>
>>>> I thought it might be convenient for testing propose, but not
>> recommended
>>>> for production,
>>>> cause Flink SQLs could be complicated and involves lots of characters
>> that
>>>> need to escape.
>>>>
>>>> WDYT?
>>>>
>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>> "run-application"
>>>>> command. Does that mean we need to perform automatic detection in
>> Flink?
>>>>
>>>> Yes! It’s like running a PyFlink job with the following command:
>>>>
>>>> ```
>>>> ./bin/flink run \
>>>> --pyModule table.word_count \
>>>> --pyFiles examples/python/table
>>>> ```
>>>>
>>>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
>>>> automatically.
>>>>
>>>>
>>>> [1]
>>>>
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>>>
>>>> Best,
>>>> Paul Lam
>>>>
>>>>> 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
>>>>>
>>>>> Thanks Paul for the proposal.
>>>>>
>>>>> +1 for this. It is valuable in improving ease of use.
>>>>>
>>>>> I have a few questions.
>>>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>>>> strong,
>>>>> the SQLDriver is fine for me)
>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>>>> prepare
>>>>> a SQL file in an image for Kubernetes application mode, which may be a
>>>> bit
>>>>> cumbersome.
>>>>> - I noticed that we don't specify the SQLDriver jar in the
>>>> "run-application"
>>>>> command. Does that mean we need to perform automatic detection in
>> Flink?
>>>>>
>>>>>
>>>>> Best,
>>>>> Weihua
>>>>>
>>>>>
>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com>
>> wrote:
>>>>>
>>>>>> Hi team,
>>>>>>
>>>>>> I’d like to start a discussion about FLIP-316 [1], which introduces a
>>>> SQL
>>>>>> driver as the
>>>>>> default main class for Flink SQL jobs.
>>>>>>
>>>>>> Currently, Flink SQL could be executed out of the box either via SQL
>>>>>> Client/Gateway
>>>>>> or embedded in a Flink Java/Python program.
>>>>>>
>>>>>> However, each one has its drawback:
>>>>>>
>>>>>> - SQL Client/Gateway doesn’t support the application deployment mode
>> [2]
>>>>>> - Flink Java/Python program requires extra work to write a non-SQL
>>>> program
>>>>>>
>>>>>> Therefore, I propose adding a SQL driver to act as the default main
>>>> class
>>>>>> for SQL jobs.
>>>>>> Please see the FLIP docs for details and feel free to comment. Thanks!
>>>>>>
>>>>>> [1]
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>>>> <
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>>>
>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>>>
>>>>>> Best,
>>>>>> Paul Lam
>>>>
>>>>
>>
>>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Weihua Hu <hu...@gmail.com>.
Thanks Paul for your reply.
SQLDriver looks good to me.
2. Do you mean a pass the SQL string a configuration or a program argument?
I brought this up because we were unable to pass the SQL file to Flink
using Kubernetes mode.
For DataStream/Python users, they need to prepare their images for the jars
and dependencies.
But for SQL users, they can use a common image to run different SQL queries
if there are no other udf requirements.
It would be great if the SQL query and image were not bound.
Using strings is a way to decouple these, but just as you mentioned, it's
not easy to pass complex SQL.
> use web submission
AFAIK, we can not use web submission in the Application mode. Please
correct me if I'm wrong.
Best,
Weihua
On Wed, May 31, 2023 at 9:37 PM Paul Lam <pa...@gmail.com> wrote:
> Hi Biao,
>
> Thanks for your comments!
>
> > 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
> in
> > Application mode? More specifically, if we use SQL client/gateway to
> > execute some interactive SQLs like a SELECT query, can we ask flink to
> use
> > Application mode to execute those queries after this FLIP?
>
> Thanks for pointing it out. I think only DMLs would be executed via SQL
> Driver.
> I'll add the scope to the FLIP.
>
> > 2. Deployment: I believe in YARN mode, the implementation is trivial as
> we
> > can ship files via YARN's tool easily but for K8s, things can be more
> > complicated as Shengkai said.
>
>
> Your input is very informative. I’m thinking about using web submission,
> but it requires exposing the JobManager port which could also be a problem
> on K8s.
>
> Another approach is to explicitly require a distributed storage to ship
> files,
> but we may need a new deployment executor for that.
>
> What do you think of these two approaches?
>
> > 3. Serialization of SessionState: in SessionState, there are some
> > unserializable fields
> > like org.apache.flink.table.resource.ResourceManager#userClassLoader. It
> > may be worthwhile to add more details about the serialization part.
>
> I agree. That’s a missing part. But if we use ExecNodeGraph as Shengkai
> mentioned, do we eliminate the need for serialization of SessionState?
>
> Best,
> Paul Lam
>
> > 2023年5月31日 13:07，Biao Geng <bi...@gmail.com> 写道：
> >
> > Thanks Paul for the proposal!I believe it would be very useful for flink
> > users.
> > After reading the FLIP, I have some questions:
> > 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs
> in
> > Application mode? More specifically, if we use SQL client/gateway to
> > execute some interactive SQLs like a SELECT query, can we ask flink to
> use
> > Application mode to execute those queries after this FLIP?
> > 2. Deployment: I believe in YARN mode, the implementation is trivial as
> we
> > can ship files via YARN's tool easily but for K8s, things can be more
> > complicated as Shengkai said. I have implemented a simple POC
> > <
> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133
> >
> > based on SQL client before(i.e. consider the SQL client which supports
> > executing a SQL file as the SQL driver in this FLIP). One problem I have
> > met is how do we ship SQL files ( or Job Graph) to the k8s side. Without
> > such support, users have to modify the initContainer or rebuild a new K8s
> > image every time to fetch the SQL file. Like the flink k8s operator, one
> > workaround is to utilize the flink config(transforming the SQL file to a
> > escaped string like Weihua mentioned) which will be converted to a
> > ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
> > <https://kubernetes.io/docs/concepts/configuration/configmap/>). Not
> sure
> > if we have better solutions.
> > 3. Serialization of SessionState: in SessionState, there are some
> > unserializable fields
> > like org.apache.flink.table.resource.ResourceManager#userClassLoader. It
> > may be worthwhile to add more details about the serialization part.
> >
> > Best,
> > Biao Geng
> >
> > Paul Lam <pa...@gmail.com> 于2023年5月31日周三 11:49写道：
> >
> >> Hi Weihua,
> >>
> >> Thanks a lot for your input! Please see my comments inline.
> >>
> >>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> >> strong,
> >>> the SQLDriver is fine for me)
> >>
> >> I’ve thought about SQL Runner but picked SQL Driver for the following
> >> reasons FYI:
> >>
> >> 1. I have a PythonDriver doing the same job for PyFlink [1]
> >> 2. Flink program's main class is sort of like Driver in JDBC which
> >> translates SQLs into
> >> databases specific languages.
> >>
> >> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> >>
> >>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
> >> prepare
> >>> a SQL file in an image for Kubernetes application mode, which may be a
> >> bit
> >>> cumbersome.
> >>
> >> Do you mean a pass the SQL string a configuration or a program argument?
> >>
> >> I thought it might be convenient for testing propose, but not
> recommended
> >> for production,
> >> cause Flink SQLs could be complicated and involves lots of characters
> that
> >> need to escape.
> >>
> >> WDYT?
> >>
> >>> - I noticed that we don't specify the SQLDriver jar in the
> >> "run-application"
> >>> command. Does that mean we need to perform automatic detection in
> Flink?
> >>
> >> Yes! It’s like running a PyFlink job with the following command:
> >>
> >> ```
> >> ./bin/flink run \
> >> --pyModule table.word_count \
> >> --pyFiles examples/python/table
> >> ```
> >>
> >> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
> >> automatically.
> >>
> >>
> >> [1]
> >>
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
> >>
> >> Best,
> >> Paul Lam
> >>
> >>> 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
> >>>
> >>> Thanks Paul for the proposal.
> >>>
> >>> +1 for this. It is valuable in improving ease of use.
> >>>
> >>> I have a few questions.
> >>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> >> strong,
> >>> the SQLDriver is fine for me)
> >>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
> >> prepare
> >>> a SQL file in an image for Kubernetes application mode, which may be a
> >> bit
> >>> cumbersome.
> >>> - I noticed that we don't specify the SQLDriver jar in the
> >> "run-application"
> >>> command. Does that mean we need to perform automatic detection in
> Flink?
> >>>
> >>>
> >>> Best,
> >>> Weihua
> >>>
> >>>
> >>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com>
> wrote:
> >>>
> >>>> Hi team,
> >>>>
> >>>> I’d like to start a discussion about FLIP-316 [1], which introduces a
> >> SQL
> >>>> driver as the
> >>>> default main class for Flink SQL jobs.
> >>>>
> >>>> Currently, Flink SQL could be executed out of the box either via SQL
> >>>> Client/Gateway
> >>>> or embedded in a Flink Java/Python program.
> >>>>
> >>>> However, each one has its drawback:
> >>>>
> >>>> - SQL Client/Gateway doesn’t support the application deployment mode
> [2]
> >>>> - Flink Java/Python program requires extra work to write a non-SQL
> >> program
> >>>>
> >>>> Therefore, I propose adding a SQL driver to act as the default main
> >> class
> >>>> for SQL jobs.
> >>>> Please see the FLIP docs for details and feel free to comment. Thanks!
> >>>>
> >>>> [1]
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >>>> <
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>>>
> >>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> >>>> https://issues.apache.org/jira/browse/FLINK-26541>
> >>>>
> >>>> Best,
> >>>> Paul Lam
> >>
> >>
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Biao,
Thanks for your comments!
> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs in
> Application mode? More specifically, if we use SQL client/gateway to
> execute some interactive SQLs like a SELECT query, can we ask flink to use
> Application mode to execute those queries after this FLIP?
Thanks for pointing it out. I think only DMLs would be executed via SQL Driver.
I'll add the scope to the FLIP.
> 2. Deployment: I believe in YARN mode, the implementation is trivial as we
> can ship files via YARN's tool easily but for K8s, things can be more
> complicated as Shengkai said.
Your input is very informative. I’m thinking about using web submission,
but it requires exposing the JobManager port which could also be a problem
on K8s.
Another approach is to explicitly require a distributed storage to ship files,
but we may need a new deployment executor for that.
What do you think of these two approaches?
> 3. Serialization of SessionState: in SessionState, there are some
> unserializable fields
> like org.apache.flink.table.resource.ResourceManager#userClassLoader. It
> may be worthwhile to add more details about the serialization part.
I agree. That’s a missing part. But if we use ExecNodeGraph as Shengkai
mentioned, do we eliminate the need for serialization of SessionState?
Best,
Paul Lam
> 2023年5月31日 13:07，Biao Geng <bi...@gmail.com> 写道：
>
> Thanks Paul for the proposal!I believe it would be very useful for flink
> users.
> After reading the FLIP, I have some questions:
> 1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs in
> Application mode? More specifically, if we use SQL client/gateway to
> execute some interactive SQLs like a SELECT query, can we ask flink to use
> Application mode to execute those queries after this FLIP?
> 2. Deployment: I believe in YARN mode, the implementation is trivial as we
> can ship files via YARN's tool easily but for K8s, things can be more
> complicated as Shengkai said. I have implemented a simple POC
> <https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
> based on SQL client before(i.e. consider the SQL client which supports
> executing a SQL file as the SQL driver in this FLIP). One problem I have
> met is how do we ship SQL files ( or Job Graph) to the k8s side. Without
> such support, users have to modify the initContainer or rebuild a new K8s
> image every time to fetch the SQL file. Like the flink k8s operator, one
> workaround is to utilize the flink config(transforming the SQL file to a
> escaped string like Weihua mentioned) which will be converted to a
> ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
> <https://kubernetes.io/docs/concepts/configuration/configmap/>). Not sure
> if we have better solutions.
> 3. Serialization of SessionState: in SessionState, there are some
> unserializable fields
> like org.apache.flink.table.resource.ResourceManager#userClassLoader. It
> may be worthwhile to add more details about the serialization part.
>
> Best,
> Biao Geng
>
> Paul Lam <pa...@gmail.com> 于2023年5月31日周三 11:49写道：
>
>> Hi Weihua,
>>
>> Thanks a lot for your input! Please see my comments inline.
>>
>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>> strong,
>>> the SQLDriver is fine for me)
>>
>> I’ve thought about SQL Runner but picked SQL Driver for the following
>> reasons FYI:
>>
>> 1. I have a PythonDriver doing the same job for PyFlink [1]
>> 2. Flink program's main class is sort of like Driver in JDBC which
>> translates SQLs into
>> databases specific languages.
>>
>> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>>
>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>> prepare
>>> a SQL file in an image for Kubernetes application mode, which may be a
>> bit
>>> cumbersome.
>>
>> Do you mean a pass the SQL string a configuration or a program argument?
>>
>> I thought it might be convenient for testing propose, but not recommended
>> for production,
>> cause Flink SQLs could be complicated and involves lots of characters that
>> need to escape.
>>
>> WDYT?
>>
>>> - I noticed that we don't specify the SQLDriver jar in the
>> "run-application"
>>> command. Does that mean we need to perform automatic detection in Flink?
>>
>> Yes! It’s like running a PyFlink job with the following command:
>>
>> ```
>> ./bin/flink run \
>> --pyModule table.word_count \
>> --pyFiles examples/python/table
>> ```
>>
>> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
>> automatically.
>>
>>
>> [1]
>> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>>
>> Best,
>> Paul Lam
>>
>>> 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
>>>
>>> Thanks Paul for the proposal.
>>>
>>> +1 for this. It is valuable in improving ease of use.
>>>
>>> I have a few questions.
>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>> strong,
>>> the SQLDriver is fine for me)
>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>> prepare
>>> a SQL file in an image for Kubernetes application mode, which may be a
>> bit
>>> cumbersome.
>>> - I noticed that we don't specify the SQLDriver jar in the
>> "run-application"
>>> command. Does that mean we need to perform automatic detection in Flink?
>>>
>>>
>>> Best,
>>> Weihua
>>>
>>>
>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com> wrote:
>>>
>>>> Hi team,
>>>>
>>>> I’d like to start a discussion about FLIP-316 [1], which introduces a
>> SQL
>>>> driver as the
>>>> default main class for Flink SQL jobs.
>>>>
>>>> Currently, Flink SQL could be executed out of the box either via SQL
>>>> Client/Gateway
>>>> or embedded in a Flink Java/Python program.
>>>>
>>>> However, each one has its drawback:
>>>>
>>>> - SQL Client/Gateway doesn’t support the application deployment mode [2]
>>>> - Flink Java/Python program requires extra work to write a non-SQL
>> program
>>>>
>>>> Therefore, I propose adding a SQL driver to act as the default main
>> class
>>>> for SQL jobs.
>>>> Please see the FLIP docs for details and feel free to comment. Thanks!
>>>>
>>>> [1]
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>> <
>>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>
>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>
>>>> Best,
>>>> Paul Lam
>>
>>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Biao Geng <bi...@gmail.com>.
Thanks Paul for the proposal!I believe it would be very useful for flink
users.
After reading the FLIP, I have some questions:
1. Scope: is this FLIP only targeted for non-interactive Flink SQL jobs in
Application mode? More specifically, if we use SQL client/gateway to
execute some interactive SQLs like a SELECT query, can we ask flink to use
Application mode to execute those queries after this FLIP?
2. Deployment: I believe in YARN mode, the implementation is trivial as we
can ship files via YARN's tool easily but for K8s, things can be more
complicated as Shengkai said. I have implemented a simple POC
<https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133>
based on SQL client before(i.e. consider the SQL client which supports
executing a SQL file as the SQL driver in this FLIP). One problem I have
met is how do we ship SQL files ( or Job Graph) to the k8s side. Without
such support, users have to modify the initContainer or rebuild a new K8s
image every time to fetch the SQL file. Like the flink k8s operator, one
workaround is to utilize the flink config(transforming the SQL file to a
escaped string like Weihua mentioned) which will be converted to a
ConfigMap but K8s has size limit of ConfigMaps(no larger than 1MB
<https://kubernetes.io/docs/concepts/configuration/configmap/>). Not sure
if we have better solutions.
3. Serialization of SessionState: in SessionState, there are some
unserializable fields
like org.apache.flink.table.resource.ResourceManager#userClassLoader. It
may be worthwhile to add more details about the serialization part.
Best,
Biao Geng
Paul Lam <pa...@gmail.com> 于2023年5月31日周三 11:49写道：
> Hi Weihua,
>
> Thanks a lot for your input! Please see my comments inline.
>
> > - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> strong,
> > the SQLDriver is fine for me)
>
> I’ve thought about SQL Runner but picked SQL Driver for the following
> reasons FYI:
>
> 1. I have a PythonDriver doing the same job for PyFlink [1]
> 2. Flink program's main class is sort of like Driver in JDBC which
> translates SQLs into
> databases specific languages.
>
> In general, I’m +1 for SQL Driver and +0 for SQL Runner.
>
> > - Could we run SQL jobs using SQL in strings? Otherwise, we need to
> prepare
> > a SQL file in an image for Kubernetes application mode, which may be a
> bit
> > cumbersome.
>
> Do you mean a pass the SQL string a configuration or a program argument?
>
> I thought it might be convenient for testing propose, but not recommended
> for production,
> cause Flink SQLs could be complicated and involves lots of characters that
> need to escape.
>
> WDYT?
>
> > - I noticed that we don't specify the SQLDriver jar in the
> "run-application"
> > command. Does that mean we need to perform automatic detection in Flink?
>
> Yes! It’s like running a PyFlink job with the following command:
>
> ```
> ./bin/flink run \
> --pyModule table.word_count \
> --pyFiles examples/python/table
> ```
>
> The CLI determines if it’s a SQL job, if yes apply the SQL Driver
> automatically.
>
>
> [1]
> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
>
> Best,
> Paul Lam
>
> > 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
> >
> > Thanks Paul for the proposal.
> >
> > +1 for this. It is valuable in improving ease of use.
> >
> > I have a few questions.
> > - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> strong,
> > the SQLDriver is fine for me)
> > - Could we run SQL jobs using SQL in strings? Otherwise, we need to
> prepare
> > a SQL file in an image for Kubernetes application mode, which may be a
> bit
> > cumbersome.
> > - I noticed that we don't specify the SQLDriver jar in the
> "run-application"
> > command. Does that mean we need to perform automatic detection in Flink?
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com> wrote:
> >
> >> Hi team,
> >>
> >> I’d like to start a discussion about FLIP-316 [1], which introduces a
> SQL
> >> driver as the
> >> default main class for Flink SQL jobs.
> >>
> >> Currently, Flink SQL could be executed out of the box either via SQL
> >> Client/Gateway
> >> or embedded in a Flink Java/Python program.
> >>
> >> However, each one has its drawback:
> >>
> >> - SQL Client/Gateway doesn’t support the application deployment mode [2]
> >> - Flink Java/Python program requires extra work to write a non-SQL
> program
> >>
> >> Therefore, I propose adding a SQL driver to act as the default main
> class
> >> for SQL jobs.
> >> Please see the FLIP docs for details and feel free to comment. Thanks!
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> >> <
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >>>
> >> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> >> https://issues.apache.org/jira/browse/FLINK-26541>
> >>
> >> Best,
> >> Paul Lam
>
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Sorry for the typo. I mean “We already have a PythonDriver doing the same job for PyFlink."
Best,
Paul Lam
> 2023年5月31日 11:49，Paul Lam <pa...@gmail.com> 写道：
>
> 1. I have a PythonDriver doing the same job for PyFlink [1]
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Weihua,
Thanks a lot for your input! Please see my comments inline.
> - Is SQLRunner the better name? We use this to run a SQL Job. (Not strong,
> the SQLDriver is fine for me)
I’ve thought about SQL Runner but picked SQL Driver for the following reasons FYI:
1. I have a PythonDriver doing the same job for PyFlink [1]
2. Flink program's main class is sort of like Driver in JDBC which translates SQLs into
databases specific languages.
In general, I’m +1 for SQL Driver and +0 for SQL Runner.
> - Could we run SQL jobs using SQL in strings? Otherwise, we need to prepare
> a SQL file in an image for Kubernetes application mode, which may be a bit
> cumbersome.
Do you mean a pass the SQL string a configuration or a program argument?
I thought it might be convenient for testing propose, but not recommended for production,
cause Flink SQLs could be complicated and involves lots of characters that need to escape.
WDYT?
> - I noticed that we don't specify the SQLDriver jar in the "run-application"
> command. Does that mean we need to perform automatic detection in Flink?
Yes! It’s like running a PyFlink job with the following command:
```
./bin/flink run \
--pyModule table.word_count \
--pyFiles examples/python/table
```
The CLI determines if it’s a SQL job, if yes apply the SQL Driver automatically.
[1] https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java
Best,
Paul Lam
> 2023年5月30日 21:56，Weihua Hu <hu...@gmail.com> 写道：
>
> Thanks Paul for the proposal.
>
> +1 for this. It is valuable in improving ease of use.
>
> I have a few questions.
> - Is SQLRunner the better name? We use this to run a SQL Job. (Not strong,
> the SQLDriver is fine for me)
> - Could we run SQL jobs using SQL in strings? Otherwise, we need to prepare
> a SQL file in an image for Kubernetes application mode, which may be a bit
> cumbersome.
> - I noticed that we don't specify the SQLDriver jar in the "run-application"
> command. Does that mean we need to perform automatic detection in Flink?
>
>
> Best,
> Weihua
>
>
> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com> wrote:
>
>> Hi team,
>>
>> I’d like to start a discussion about FLIP-316 [1], which introduces a SQL
>> driver as the
>> default main class for Flink SQL jobs.
>>
>> Currently, Flink SQL could be executed out of the box either via SQL
>> Client/Gateway
>> or embedded in a Flink Java/Python program.
>>
>> However, each one has its drawback:
>>
>> - SQL Client/Gateway doesn’t support the application deployment mode [2]
>> - Flink Java/Python program requires extra work to write a non-SQL program
>>
>> Therefore, I propose adding a SQL driver to act as the default main class
>> for SQL jobs.
>> Please see the FLIP docs for details and feel free to comment. Thanks!
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>> <
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>
>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>> https://issues.apache.org/jira/browse/FLINK-26541>
>>
>> Best,
>> Paul Lam
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Shengkai,
Thanks a lot for your comments! Please see my comments inline.
> 1. The FLIP does not specify the kind of SQL that will be submitted with
> the application mode. I believe only a portion of the SQL will be delegated
> to the SqlRunner.
You’re right. For SQL Gateway, only DMLs need to be delegated to the SQL
Driver. I would think about the details and update the FLIP. Do you have some
ideas already?
> 2. Will the SQL Client/Gateway perform any validation before submitting the
> SQL to the SqlRunner? If the user's SQL is invalid, it could take a long
> time to fetch error messages before execution.
Yes, I think so. We could introduce a SQL Gateway option like
`sql-gateway.application.validate-sql`, which determines if the Gateway
should do the SQL validation by compiling the plan.
> 3. ExecNodeGraph VS SQL File
ExecNodeGraph looks good to me. It’s not efficient to rebuild the
same TableEnvironment from scratch with SQL files at JobManager.
ExecNodeGraph avoids that.
Then I think SQL Driver should accept SQL files and ExecNodeGraph,
while the former is for public usage and the latter is for internal usage.
> 4. SqlRunner VS ApplicationRunner
ApplicationRunner looks good for SQL Driver at first glance, but
I’m afraid it has two limitations:
1) IIUC, ApplicationRunner should only be used in application mode, but
SQL Driver actually could be used in other deployment modes as well.
2) As FLIP-85 said, ApplicationRunner doesn’t work well with YARN.
What do you think?
> 5. K8S Application mode
WRT K8s application mode, I think it’s more of the users’ responsibility to
prepare the basic resources by customizing the image.
For non-SQL-gateway scenarios, the image should contain everything
needed for the SQLs.
For SQL-gateway scenarios, that’s actually a missing part that we need to
discuss. The gateway-generated resources (e.g. ExecNodeGraph) can’t be
accessed at JobManager, unless:
1) use web submission to deploy and run jars
2) mount a distributed storage to both SQL gateway and all nodes that
Flink runs on
I lean toward the web submission way but didn’t think through the details yet.
BTW, it seems that we don’t need `pipeline.jars` on K8s, we could put the
jars into the classpath directly.
> 6. Could you add more details about your modification in the gateway side,
> including error handling, execution workflow, and the impact on job
> statements?
Sure. I’ll let you know when I finish the details.
Paul Lam
> 2023年5月31日 10:56，Shengkai Fang <fs...@gmail.com> 写道：
>
> Thanks for the proposal. The Application mode is very important to Flink
> SQL. But I have some questions about the FLIP:
>
> 1. The FLIP does not specify the kind of SQL that will be submitted with
> the application mode. I believe only a portion of the SQL will be delegated
> to the SqlRunner.
> 2. Will the SQL Client/Gateway perform any validation before submitting the
> SQL to the SqlRunner? If the user's SQL is invalid, it could take a long
> time to fetch error messages before execution.
> 3. ExecNodeGraph VS SQL File
>
> Initially, we planned to use ExecNodeGraph as the essential information to
> be submitted. By enabling 'table.plan.compile.catalog-objects' = 'all', the
> ExecNodeGraph provides necessary information about the session state.
> ExecNodeGraph contains the basic structure of tables and the class name of
> catalog functions used, which enables us to avoid serializing the catalog
> to the remote. Therefore, we prefer to use ExecGraph as the content
> submitted. Furthermore, our internal implementation can extend ExecGraph
> beyond state compatibility.
>
> 4. SqlRunner VS ApplicationRunner
>
> In the FLIP-85, it mentions to support Library mode. Compared to adding a
> new module, I think it's better to extend the origin design? WDYT?
>
> ApplicationRunner.run((StreamExecutionEnvironment env) -> {
>
> … // a consumer of the env
>
> })
>
> 5. K8S Application mode
>
> As far as I know, K8S doesn't support shipping multiple jars to the remote.
> It seems the current design also doesn't support K8S Application mode?
>
> https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L217
>
> 6. Could you add more details about your modification in the gateway side,
> including error handling, execution workflow, and the impact on job
> statements?
>
> Best,
> Shengkai
>
> Shammon FY <zj...@gmail.com> 于2023年5月31日周三 10:40写道：
>
>> Thanks Paul for driving this proposal.
>>
>> I found the sql driver has no config related options. If I understand
>> correctly, the sql driver can be used to submit sql jobs in a 'job
>> submission service' such as sql-gateway. In general, in addition to the
>> default config for Flink cluster which includes k8s, ha and .etc, users may
>> also specify configurations for SQL jobs, including parallelism, number of
>> Task Managers, etc.
>>
>> For example, in sql-gateway users can `set` dynamic parameters before
>> submitting a sql statement, and for sql files users may put their
>> configurations in a `yaml` file. Should sql driver support dynamic
>> parameters and specify config file?
>>
>> Best,
>> Shammon FY
>>
>> On Tue, May 30, 2023 at 9:57 PM Weihua Hu <hu...@gmail.com> wrote:
>>
>>> Thanks Paul for the proposal.
>>>
>>> +1 for this. It is valuable in improving ease of use.
>>>
>>> I have a few questions.
>>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not
>> strong,
>>> the SQLDriver is fine for me)
>>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to
>> prepare
>>> a SQL file in an image for Kubernetes application mode, which may be a
>> bit
>>> cumbersome.
>>> - I noticed that we don't specify the SQLDriver jar in the
>>> "run-application"
>>> command. Does that mean we need to perform automatic detection in Flink?
>>>
>>>
>>> Best,
>>> Weihua
>>>
>>>
>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com> wrote:
>>>
>>>> Hi team,
>>>>
>>>> I’d like to start a discussion about FLIP-316 [1], which introduces a
>> SQL
>>>> driver as the
>>>> default main class for Flink SQL jobs.
>>>>
>>>> Currently, Flink SQL could be executed out of the box either via SQL
>>>> Client/Gateway
>>>> or embedded in a Flink Java/Python program.
>>>>
>>>> However, each one has its drawback:
>>>>
>>>> - SQL Client/Gateway doesn’t support the application deployment mode
>> [2]
>>>> - Flink Java/Python program requires extra work to write a non-SQL
>>> program
>>>>
>>>> Therefore, I propose adding a SQL driver to act as the default main
>> class
>>>> for SQL jobs.
>>>> Please see the FLIP docs for details and feel free to comment. Thanks!
>>>>
>>>> [1]
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>>> <
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>>
>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>>
>>>> Best,
>>>> Paul Lam
>>>
>>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Shengkai Fang <fs...@gmail.com>.
Thanks for the proposal. The Application mode is very important to Flink
SQL. But I have some questions about the FLIP:
1. The FLIP does not specify the kind of SQL that will be submitted with
the application mode. I believe only a portion of the SQL will be delegated
to the SqlRunner.
2. Will the SQL Client/Gateway perform any validation before submitting the
SQL to the SqlRunner? If the user's SQL is invalid, it could take a long
time to fetch error messages before execution.
3. ExecNodeGraph VS SQL File
Initially, we planned to use ExecNodeGraph as the essential information to
be submitted. By enabling 'table.plan.compile.catalog-objects' = 'all', the
ExecNodeGraph provides necessary information about the session state.
ExecNodeGraph contains the basic structure of tables and the class name of
catalog functions used, which enables us to avoid serializing the catalog
to the remote. Therefore, we prefer to use ExecGraph as the content
submitted. Furthermore, our internal implementation can extend ExecGraph
beyond state compatibility.
4. SqlRunner VS ApplicationRunner
In the FLIP-85, it mentions to support Library mode. Compared to adding a
new module, I think it's better to extend the origin design? WDYT?
ApplicationRunner.run((StreamExecutionEnvironment env) -> {
… // a consumer of the env
})
5. K8S Application mode
As far as I know, K8S doesn't support shipping multiple jars to the remote.
It seems the current design also doesn't support K8S Application mode?
https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L217
6. Could you add more details about your modification in the gateway side,
including error handling, execution workflow, and the impact on job
statements?
Best,
Shengkai
Shammon FY <zj...@gmail.com> 于2023年5月31日周三 10:40写道：
> Thanks Paul for driving this proposal.
>
> I found the sql driver has no config related options. If I understand
> correctly, the sql driver can be used to submit sql jobs in a 'job
> submission service' such as sql-gateway. In general, in addition to the
> default config for Flink cluster which includes k8s, ha and .etc, users may
> also specify configurations for SQL jobs, including parallelism, number of
> Task Managers, etc.
>
> For example, in sql-gateway users can `set` dynamic parameters before
> submitting a sql statement, and for sql files users may put their
> configurations in a `yaml` file. Should sql driver support dynamic
> parameters and specify config file?
>
> Best,
> Shammon FY
>
> On Tue, May 30, 2023 at 9:57 PM Weihua Hu <hu...@gmail.com> wrote:
>
> > Thanks Paul for the proposal.
> >
> > +1 for this. It is valuable in improving ease of use.
> >
> > I have a few questions.
> > - Is SQLRunner the better name? We use this to run a SQL Job. (Not
> strong,
> > the SQLDriver is fine for me)
> > - Could we run SQL jobs using SQL in strings? Otherwise, we need to
> prepare
> > a SQL file in an image for Kubernetes application mode, which may be a
> bit
> > cumbersome.
> > - I noticed that we don't specify the SQLDriver jar in the
> > "run-application"
> > command. Does that mean we need to perform automatic detection in Flink?
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com> wrote:
> >
> > > Hi team,
> > >
> > > I’d like to start a discussion about FLIP-316 [1], which introduces a
> SQL
> > > driver as the
> > > default main class for Flink SQL jobs.
> > >
> > > Currently, Flink SQL could be executed out of the box either via SQL
> > > Client/Gateway
> > > or embedded in a Flink Java/Python program.
> > >
> > > However, each one has its drawback:
> > >
> > > - SQL Client/Gateway doesn’t support the application deployment mode
> [2]
> > > - Flink Java/Python program requires extra work to write a non-SQL
> > program
> > >
> > > Therefore, I propose adding a SQL driver to act as the default main
> class
> > > for SQL jobs.
> > > Please see the FLIP docs for details and feel free to comment. Thanks!
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > > >
> > > [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> > > https://issues.apache.org/jira/browse/FLINK-26541>
> > >
> > > Best,
> > > Paul Lam
> >
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Paul Lam <pa...@gmail.com>.
Hi Shammon,
Thanks a lot for your input!
I thought SQL Driver could act as a general-purpose default main class
for Flink SQL. It could be used in Flink CLI submission, web submission,
or SQL Client/Gateway submission. For SQL Client/Gateway submission,
we use it implicitly if needed, and for the other approaches, explicitly.
WRT to configurations, dynamic configurations are supported for most
cases ATM.
- For CLI submission, the dynamic configuration is supported via CLI,
i.e. `-D` prefixed configurations.
- For SQL Client/Gateway submission, we could also use `-D` dynamic
configuration and SQL init file to customize the default session configuration.
- For web submission, there’s no dynamic configuration as I’m aware of.
Do you think it’s sufficient already or is it still necessary to implement it at
the SQL driver level?
Best,
Paul Lam
> 2023年5月31日 10:40，Shammon FY <zj...@gmail.com> 写道：
>
> Thanks Paul for driving this proposal.
>
> I found the sql driver has no config related options. If I understand
> correctly, the sql driver can be used to submit sql jobs in a 'job
> submission service' such as sql-gateway. In general, in addition to the
> default config for Flink cluster which includes k8s, ha and .etc, users may
> also specify configurations for SQL jobs, including parallelism, number of
> Task Managers, etc.
>
> For example, in sql-gateway users can `set` dynamic parameters before
> submitting a sql statement, and for sql files users may put their
> configurations in a `yaml` file. Should sql driver support dynamic
> parameters and specify config file?
>
> Best,
> Shammon FY
>
> On Tue, May 30, 2023 at 9:57 PM Weihua Hu <hu...@gmail.com> wrote:
>
>> Thanks Paul for the proposal.
>>
>> +1 for this. It is valuable in improving ease of use.
>>
>> I have a few questions.
>> - Is SQLRunner the better name? We use this to run a SQL Job. (Not strong,
>> the SQLDriver is fine for me)
>> - Could we run SQL jobs using SQL in strings? Otherwise, we need to prepare
>> a SQL file in an image for Kubernetes application mode, which may be a bit
>> cumbersome.
>> - I noticed that we don't specify the SQLDriver jar in the
>> "run-application"
>> command. Does that mean we need to perform automatic detection in Flink?
>>
>>
>> Best,
>> Weihua
>>
>>
>> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com> wrote:
>>
>>> Hi team,
>>>
>>> I’d like to start a discussion about FLIP-316 [1], which introduces a SQL
>>> driver as the
>>> default main class for Flink SQL jobs.
>>>
>>> Currently, Flink SQL could be executed out of the box either via SQL
>>> Client/Gateway
>>> or embedded in a Flink Java/Python program.
>>>
>>> However, each one has its drawback:
>>>
>>> - SQL Client/Gateway doesn’t support the application deployment mode [2]
>>> - Flink Java/Python program requires extra work to write a non-SQL
>> program
>>>
>>> Therefore, I propose adding a SQL driver to act as the default main class
>>> for SQL jobs.
>>> Please see the FLIP docs for details and feel free to comment. Thanks!
>>>
>>> [1]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
>>> <
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
>>>>
>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
>>> https://issues.apache.org/jira/browse/FLINK-26541>
>>>
>>> Best,
>>> Paul Lam
>>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Shammon FY <zj...@gmail.com>.
Thanks Paul for driving this proposal.
I found the sql driver has no config related options. If I understand
correctly, the sql driver can be used to submit sql jobs in a 'job
submission service' such as sql-gateway. In general, in addition to the
default config for Flink cluster which includes k8s, ha and .etc, users may
also specify configurations for SQL jobs, including parallelism, number of
Task Managers, etc.
For example, in sql-gateway users can `set` dynamic parameters before
submitting a sql statement, and for sql files users may put their
configurations in a `yaml` file. Should sql driver support dynamic
parameters and specify config file?
Best,
Shammon FY
On Tue, May 30, 2023 at 9:57 PM Weihua Hu <hu...@gmail.com> wrote:
> Thanks Paul for the proposal.
>
> +1 for this. It is valuable in improving ease of use.
>
> I have a few questions.
> - Is SQLRunner the better name? We use this to run a SQL Job. (Not strong,
> the SQLDriver is fine for me)
> - Could we run SQL jobs using SQL in strings? Otherwise, we need to prepare
> a SQL file in an image for Kubernetes application mode, which may be a bit
> cumbersome.
> - I noticed that we don't specify the SQLDriver jar in the
> "run-application"
> command. Does that mean we need to perform automatic detection in Flink?
>
>
> Best,
> Weihua
>
>
> On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com> wrote:
>
> > Hi team,
> >
> > I’d like to start a discussion about FLIP-316 [1], which introduces a SQL
> > driver as the
> > default main class for Flink SQL jobs.
> >
> > Currently, Flink SQL could be executed out of the box either via SQL
> > Client/Gateway
> > or embedded in a Flink Java/Python program.
> >
> > However, each one has its drawback:
> >
> > - SQL Client/Gateway doesn’t support the application deployment mode [2]
> > - Flink Java/Python program requires extra work to write a non-SQL
> program
> >
> > Therefore, I propose adding a SQL driver to act as the default main class
> > for SQL jobs.
> > Please see the FLIP docs for details and feel free to comment. Thanks!
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> > >
> > [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> > https://issues.apache.org/jira/browse/FLINK-26541>
> >
> > Best,
> > Paul Lam
>
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Posted by Weihua Hu <hu...@gmail.com>.
Thanks Paul for the proposal.
+1 for this. It is valuable in improving ease of use.
I have a few questions.
- Is SQLRunner the better name? We use this to run a SQL Job. (Not strong,
the SQLDriver is fine for me)
- Could we run SQL jobs using SQL in strings? Otherwise, we need to prepare
a SQL file in an image for Kubernetes application mode, which may be a bit
cumbersome.
- I noticed that we don't specify the SQLDriver jar in the "run-application"
command. Does that mean we need to perform automatic detection in Flink?
Best,
Weihua
On Mon, May 29, 2023 at 7:24 PM Paul Lam <pa...@gmail.com> wrote:
> Hi team,
>
> I’d like to start a discussion about FLIP-316 [1], which introduces a SQL
> driver as the
> default main class for Flink SQL jobs.
>
> Currently, Flink SQL could be executed out of the box either via SQL
> Client/Gateway
> or embedded in a Flink Java/Python program.
>
> However, each one has its drawback:
>
> - SQL Client/Gateway doesn’t support the application deployment mode [2]
> - Flink Java/Python program requires extra work to write a non-SQL program
>
> Therefore, I propose adding a SQL driver to act as the default main class
> for SQL jobs.
> Please see the FLIP docs for details and feel free to comment. Thanks!
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver
> >
> [2] https://issues.apache.org/jira/browse/FLINK-26541 <
> https://issues.apache.org/jira/browse/FLINK-26541>
>
> Best,
> Paul Lam