You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Qing Lim <q....@mwam.com> on 2022/06/22 09:11:21 UTC
How can I convert a SQL String to a ResolvedExpression?
Hi Flink User Group,
Is there a way to convert SQL String to a org.apache.flink.table.expressions.ResolvedExpression easily? I wish use this in test. I manage to reverse engineer part of it but I don't think it matches what Flink is really doing, below is my ugly code.
private List<ResolvedExpression> resolveSQLFilterToExpression(
String sqlExp, ResolvedSchema schema) {
StreamTableEnvironmentImpl tbImpl = (StreamTableEnvironmentImpl) tEnv;
CatalogManager catMan = ((StreamTableEnvironmentImpl) tEnv).getCatalogManager();
FunctionCatalog funCat = new FunctionCatalog(tEnv.getConfig(), catMan, new ModuleManager());
RowType sourceType = (RowType) schema.toSourceRowDataType().getLogicalType();
RexNodeToExpressionConverter converter =
new RexNodeToExpressionConverter(
new RexBuilder(FlinkTypeFactory.INSTANCE()),
sourceType.getFieldNames().toArray(new String[0]),
funCat,
catMan,
TimeZone.getTimeZone(tEnv.getConfig().getLocalTimeZone()));
RexNodeExpression rexExp =
(RexNodeExpression) tbImpl.getParser().parseSqlExpression(sqlExp, sourceType, null);
if (rexExp.getRexNode().accept(converter).isEmpty()) {
System.out.println(rexExp);
}
ResolvedExpression resolvedExp = rexExp.getRexNode().accept(converter).get();
ExpressionResolver resolver =
ExpressionResolver.resolverFor(
tEnv.getConfig(),
name -> Optional.empty(),
new FunctionCatalog(tEnv.getConfig(), catMan, new ModuleManager())
.asLookup(
str -> {
throw new TableException(
"We should not need to lookup any expressions at this point");
}),
catMan.getDataTypeFactory(),
(sqlExpression, inputRowType, outputType) -> {
throw new TableException(
"SQL expression parsing is not supported at this location.");
})
.build();
return resolver.resolve(List.of(resolvedExp));
}
I wonder if there's some hooks available to achieve this?
Thanks
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession.
To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.
To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email.
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. (“MWNA”), which is registered with the US Securities and Exchange Commission (“SEC”) as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.
Re: Re:How can I convert a SQL String to a ResolvedExpression?
Posted by Shengkai Fang <fs...@gmail.com>.
Hi.
I think you can use Expressions#callSql to convert the String to
Expression. Then you can use ExpressionResolver to resolve the converted
Expression.
Best,
Shengkai
Qing Lim <q....@mwam.com> 于2022年6月22日周三 23:58写道：
> Hi Xuyang,
>
>
>
> Thanks for the pointer, however it does not seems to achieve what I want.
>
>
>
> I would like to provide the input in String, instead of Expression, and it
> would be best if I can reuse how flink does it.
>
>
>
> For example, given a String: “col_a = 20 AND col_b < 10”, is there any
> flink’s api that can convert it into ResolvedExpression?
>
>
>
> I manage to find a way to perform this conversion, but it is not exactly
> the same as how Flink does it, I realize this because when I try to use
> TO_TIMESTAMP function in my input string, it failed to resolve.
>
>
>
> *From:* Xuyang <xy...@163.com>
> *Sent:* 22 June 2022 16:16
> *To:* Qing Lim <q....@mwam.com>
> *Cc:* User <us...@flink.apache.org>
> *Subject:* Re:How can I convert a SQL String to a ResolvedExpression?
>
>
>
> Hi, what about use `FieldReferenceExpression`[1] in test just like here[2]
> ?
>
> [1]
> https://github.com/apache/flink/blob/5bcef81356f965ee9e6a8ab54b5faca1e3979873/flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/FieldReferenceExpression.java#L39
> [2]
> https://github.com/apache/flink/blob/5bcef81356f965ee9e6a8ab54b5faca1e3979873/flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/expressions/resolver/ExpressionResolverTest.java#L87
>
> At 2022-06-22 17:11:21, "Qing Lim" <q....@mwam.com> wrote:
>
> Hi Flink User Group,
>
>
>
> Is there a way to convert SQL String to a
> org.apache.flink.table.expressions.ResolvedExpression easily? I wish use
> this in test. I manage to reverse engineer part of it but I don’t think it
> matches what Flink is really doing, below is my ugly code.
>
>
>
> private List<ResolvedExpression> resolveSQLFilterToExpression(
>
> String sqlExp, ResolvedSchema schema) {
>
> StreamTableEnvironmentImpl tbImpl = (StreamTableEnvironmentImpl)
> tEnv;
>
> CatalogManager catMan = ((StreamTableEnvironmentImpl)
> tEnv).getCatalogManager();
>
> FunctionCatalog funCat = new FunctionCatalog(tEnv.getConfig(),
> catMan, new ModuleManager());
>
> RowType sourceType = (RowType)
> schema.toSourceRowDataType().getLogicalType();
>
> RexNodeToExpressionConverter converter =
>
> new RexNodeToExpressionConverter(
>
> new RexBuilder(FlinkTypeFactory.INSTANCE()),
>
> sourceType.getFieldNames().toArray(new String[0]),
>
> funCat,
>
> catMan,
>
>
> TimeZone.getTimeZone(tEnv.getConfig().getLocalTimeZone()));
>
>
>
> RexNodeExpression rexExp =
>
> (RexNodeExpression)
> tbImpl.getParser().parseSqlExpression(sqlExp, sourceType, null);
>
>
>
> if (rexExp.getRexNode().accept(converter).isEmpty()) {
>
> System.out.println(rexExp);
>
> }
>
> ResolvedExpression resolvedExp =
> rexExp.getRexNode().accept(converter).get();
>
> ExpressionResolver resolver =
>
> ExpressionResolver.resolverFor(
>
> tEnv.getConfig(),
>
> name -> Optional.empty(),
>
> new FunctionCatalog(tEnv.getConfig(),
> catMan, new ModuleManager())
>
> .asLookup(
>
> str -> {
>
> throw new
> TableException(
>
> "We should not
> need to lookup any expressions at this point");
>
> }),
>
> catMan.getDataTypeFactory(),
>
> (sqlExpression, inputRowType, outputType)
> -> {
>
> throw new TableException(
>
> "SQL expression parsing is not
> supported at this location.");
>
> })
>
> .build();
>
>
>
> return resolver.resolve(List.of(resolvedExp));
>
> }
>
>
>
> I wonder if there’s some hooks available to achieve this?
>
>
>
> Thanks
>
>
>
> This e-mail and any attachments are confidential to the addressee(s) and
> may contain information that is legally privileged and/or confidential. If
> you are not the intended recipient of this e-mail you are hereby notified
> that any dissemination, distribution, or copying of its content is strictly
> prohibited. If you have received this message in error, please notify the
> sender by return e-mail and destroy the message and all copies in your
> possession.
>
>
> To find out more details about how we may collect, use and share your
> personal information, please see https://www.mwam.com/privacy-policy.
> This includes details of how calls you make to us may be recorded in order
> for us to comply with our legal and regulatory obligations.
>
>
> To the extent that the contents of this email constitutes a financial
> promotion, please note that it is issued only to and/or directed only at
> persons who are professional clients or eligible counterparties as defined
> in the FCA Rules. Any investment products or services described in this
> email are available only to professional clients and eligible
> counterparties. Persons who are not professional clients or eligible
> counterparties should not rely or act on the contents of this email.
>
>
> Marshall Wace LLP is authorised and regulated by the Financial Conduct
> Authority. Marshall Wace LLP is a limited liability partnership registered
> in England and Wales with registered number OC302228 and registered office
> at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving
> this e-mail as a client, or an investor in an investment vehicle, managed
> or advised by Marshall Wace North America L.P., the sender of this e-mail
> is communicating with you in the sender's capacity as an associated or
> related person of Marshall Wace North America L.P. ("MWNA"), which is
> registered with the US Securities and Exchange Commission ("SEC") as an
> investment adviser. Registration with the SEC does not imply that MWNA or
> its employees possess a certain level of skill or training.
>
>
RE: Re:How can I convert a SQL String to a ResolvedExpression?
Posted by Qing Lim <q....@mwam.com>.
Hi Xuyang,
Thanks for the pointer, however it does not seems to achieve what I want.
I would like to provide the input in String, instead of Expression, and it would be best if I can reuse how flink does it.
For example, given a String: "col_a = 20 AND col_b < 10", is there any flink's api that can convert it into ResolvedExpression?
I manage to find a way to perform this conversion, but it is not exactly the same as how Flink does it, I realize this because when I try to use TO_TIMESTAMP function in my input string, it failed to resolve.
From: Xuyang <xy...@163.com>
Sent: 22 June 2022 16:16
To: Qing Lim <q....@mwam.com>
Cc: User <us...@flink.apache.org>
Subject: Re:How can I convert a SQL String to a ResolvedExpression?
Hi, what about use `FieldReferenceExpression`[1] in test just like here[2] ?
[1] https://github.com/apache/flink/blob/5bcef81356f965ee9e6a8ab54b5faca1e3979873/flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/FieldReferenceExpression.java#L39
[2] https://github.com/apache/flink/blob/5bcef81356f965ee9e6a8ab54b5faca1e3979873/flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/expressions/resolver/ExpressionResolverTest.java#L87
At 2022-06-22 17:11:21, "Qing Lim" <q....@mwam.com>> wrote:
Hi Flink User Group,
Is there a way to convert SQL String to a org.apache.flink.table.expressions.ResolvedExpression easily? I wish use this in test. I manage to reverse engineer part of it but I don't think it matches what Flink is really doing, below is my ugly code.
private List<ResolvedExpression> resolveSQLFilterToExpression(
String sqlExp, ResolvedSchema schema) {
StreamTableEnvironmentImpl tbImpl = (StreamTableEnvironmentImpl) tEnv;
CatalogManager catMan = ((StreamTableEnvironmentImpl) tEnv).getCatalogManager();
FunctionCatalog funCat = new FunctionCatalog(tEnv.getConfig(), catMan, new ModuleManager());
RowType sourceType = (RowType) schema.toSourceRowDataType().getLogicalType();
RexNodeToExpressionConverter converter =
new RexNodeToExpressionConverter(
new RexBuilder(FlinkTypeFactory.INSTANCE()),
sourceType.getFieldNames().toArray(new String[0]),
funCat,
catMan,
TimeZone.getTimeZone(tEnv.getConfig().getLocalTimeZone()));
RexNodeExpression rexExp =
(RexNodeExpression) tbImpl.getParser().parseSqlExpression(sqlExp, sourceType, null);
if (rexExp.getRexNode().accept(converter).isEmpty()) {
System.out.println(rexExp);
}
ResolvedExpression resolvedExp = rexExp.getRexNode().accept(converter).get();
ExpressionResolver resolver =
ExpressionResolver.resolverFor(
tEnv.getConfig(),
name -> Optional.empty(),
new FunctionCatalog(tEnv.getConfig(), catMan, new ModuleManager())
.asLookup(
str -> {
throw new TableException(
"We should not need to lookup any expressions at this point");
}),
catMan.getDataTypeFactory(),
(sqlExpression, inputRowType, outputType) -> {
throw new TableException(
"SQL expression parsing is not supported at this location.");
})
.build();
return resolver.resolve(List.of(resolvedExp));
}
I wonder if there's some hooks available to achieve this?
Thanks
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession.
To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.
To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email.
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. ("MWNA"), which is registered with the US Securities and Exchange Commission ("SEC") as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.
Re:How can I convert a SQL String to a ResolvedExpression?
Posted by Xuyang <xy...@163.com>.
Hi, what about use `FieldReferenceExpression`[1] in test just like here[2] ?
[1] https://github.com/apache/flink/blob/5bcef81356f965ee9e6a8ab54b5faca1e3979873/flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/FieldReferenceExpression.java#L39
[2] https://github.com/apache/flink/blob/5bcef81356f965ee9e6a8ab54b5faca1e3979873/flink-table/flink-table-api-java/src/test/java/org/apache/flink/table/expressions/resolver/ExpressionResolverTest.java#L87
At 2022-06-22 17:11:21, "Qing Lim" <q....@mwam.com> wrote:
Hi Flink User Group,
Is there a way to convert SQL String to a org.apache.flink.table.expressions.ResolvedExpression easily? I wish use this in test. I manage to reverse engineer part of it but I don’t think it matches what Flink is really doing, below is my ugly code.
private List<ResolvedExpression> resolveSQLFilterToExpression(
String sqlExp, ResolvedSchema schema) {
StreamTableEnvironmentImpl tbImpl = (StreamTableEnvironmentImpl) tEnv;
CatalogManager catMan = ((StreamTableEnvironmentImpl) tEnv).getCatalogManager();
FunctionCatalog funCat = new FunctionCatalog(tEnv.getConfig(), catMan, new ModuleManager());
RowType sourceType = (RowType) schema.toSourceRowDataType().getLogicalType();
RexNodeToExpressionConverter converter =
new RexNodeToExpressionConverter(
new RexBuilder(FlinkTypeFactory.INSTANCE()),
sourceType.getFieldNames().toArray(new String[0]),
funCat,
catMan,
TimeZone.getTimeZone(tEnv.getConfig().getLocalTimeZone()));
RexNodeExpression rexExp =
(RexNodeExpression) tbImpl.getParser().parseSqlExpression(sqlExp, sourceType, null);
if (rexExp.getRexNode().accept(converter).isEmpty()) {
System.out.println(rexExp);
}
ResolvedExpression resolvedExp = rexExp.getRexNode().accept(converter).get();
ExpressionResolver resolver =
ExpressionResolver.resolverFor(
tEnv.getConfig(),
name -> Optional.empty(),
new FunctionCatalog(tEnv.getConfig(), catMan, new ModuleManager())
.asLookup(
str -> {
throw new TableException(
"We should not need to lookup any expressions at this point");
}),
catMan.getDataTypeFactory(),
(sqlExpression, inputRowType, outputType) -> {
throw new TableException(
"SQL expression parsing is not supported at this location.");
})
.build();
return resolver.resolve(List.of(resolvedExp));
}
I wonder if there’s some hooks available to achieve this?
Thanks
This e-mail and any attachments are confidential to the addressee(s) and may contain information that is legally privileged and/or confidential. If you are not the intended recipient of this e-mail you are hereby notified that any dissemination, distribution, or copying of its content is strictly prohibited. If you have received this message in error, please notify the sender by return e-mail and destroy the message and all copies in your possession.
To find out more details about how we may collect, use and share your personal information, please see https://www.mwam.com/privacy-policy. This includes details of how calls you make to us may be recorded in order for us to comply with our legal and regulatory obligations.
To the extent that the contents of this email constitutes a financial promotion, please note that it is issued only to and/or directed only at persons who are professional clients or eligible counterparties as defined in the FCA Rules. Any investment products or services described in this email are available only to professional clients and eligible counterparties. Persons who are not professional clients or eligible counterparties should not rely or act on the contents of this email.
Marshall Wace LLP is authorised and regulated by the Financial Conduct Authority. Marshall Wace LLP is a limited liability partnership registered in England and Wales with registered number OC302228 and registered office at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving this e-mail as a client, or an investor in an investment vehicle, managed or advised by Marshall Wace North America L.P., the sender of this e-mail is communicating with you in the sender's capacity as an associated or related person of Marshall Wace North America L.P. ("MWNA"), which is registered with the US Securities and Exchange Commission ("SEC") as an investment adviser. Registration with the SEC does not imply that MWNA or its employees possess a certain level of skill or training.