You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "superdiaodiao (via GitHub)" <gi...@apache.org> on 2024/05/12 13:23:21 UTC
[PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
superdiaodiao opened a new pull request, #24773:
URL: https://github.com/apache/flink/pull/24773
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
This is an implementation of URL_ENCODE and URL_DECODE
1. URL_ENCODE: Translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme(UTF-8).
2. URL_DECODE: Decodes a string in 'application/x-www-form-urlencoded' format using a specific encoding scheme(UTF-8).
## Brief change log
1. **URL_ENCODE**
- Syntax:
url_encode(url)
- Arguments:
url: a string represents a URL
- Returns:
translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme(UTF-8), will be null if input is null.
- Examples:
```
url = 'https://flink.apache.org/'
SQL: url_encode(url)
TableAPI: url.urlEncode()
output: 'https%3A%2F%2Fflink.apache.org%2F'
```
2. **URL_DECODE**
- Syntax:
url_decode(value)
- Arguments:
value: a URL encoded
- Returns:
decodes a string in 'application/x-www-form-urlencoded' format using a specific encoding scheme(UTF-8), will be null if input is null.
- Examples:
```
value = 'https%3A%2F%2Fflink.apache.org%2F'
SQL: url_decode(value)
TableAPI: value.urlDecode()
output: 'https://flink.apache.org/'
```
## Verifying this change
- This change added tests in UrlFunctionITCase.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606752091
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlEncodeFunction.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_ENCODE}. */
+@Internal
+public class UrlEncodeFunction extends BuiltInScalarFunction {
+
+ public UrlEncodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_ENCODE, context);
+ }
+
+ public @Nullable StringData eval(StringData url) {
+ if (url == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ String value;
+
+ try {
+ value = URLEncoder.encode(url.toString(), charset.name());
+ } catch (UnsupportedEncodingException e) {
+ throw new FlinkRuntimeException(
+ "Failed to encode URL: " + url + " with charset: " + charset.name(), e);
Review Comment:
Nit:
I suggest changing the Exception message to a little more explicit "...with charset:" to .. due to an unsupported encoding with charset:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "lincoln-lil (via GitHub)" <gi...@apache.org>.
lincoln-lil commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1674142139
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java:
##########
@@ -431,6 +431,26 @@ ANY, and(logical(LogicalTypeRoot.BOOLEAN), LITERAL)
.runtimeClass("org.apache.flink.table.runtime.functions.scalar.SplitFunction")
.build();
+ public static final BuiltInFunctionDefinition URL_DECODE =
+ BuiltInFunctionDefinition.newBuilder()
+ .name("url_decode")
+ .kind(SCALAR)
+ .inputTypeStrategy(sequence(logical(LogicalTypeFamily.CHARACTER_STRING)))
+ .outputTypeStrategy(nullableIfArgs(explicit(DataTypes.STRING())))
Review Comment:
@snuyanzin Sorry I missed the ping here! IIUC the `AbstractDataType`#`nullable` says it's the default behavior means a datatype which doesn't explictly declares its nullabilty then the default behavior is nullable.
For the output type strategy, I think I should point it out more clearly: it should be 'outputTypeStrategy(explicit(DataTypes.STRING().nullable()))' because the return value may be null even if the input is non-null (e.g., decoding failure), WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2106248298
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1603092403
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ @Override
+ Stream<TestSetSpec> getTestSetSpecs() {
+
+ return Stream.of(
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
+ .onFieldsWithData(
+ "https%3A%2F%2Fflink.apache.org%2F",
+ "https://flink.apache.org/",
+ "http%3A%2F%2Ftest%3Fa%3Db%26c%3Dd",
+ null,
+ "inva+lid%3A%2F%2Fuser%3Apass%40host%2Ffile%3Bparam%3Fquery%3Bp2",
+ "")
+ .andDataTypes(STRING(), STRING(), STRING(), STRING(), STRING(), STRING())
+ .testResult(
+ $("f0").urlDecode(),
+ "url_decode(f0)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f1").urlDecode(),
+ "url_decode(f1)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f2").urlDecode(),
+ "url_decode(f2)",
+ "http://test?a=b&c=d",
+ STRING())
+ .testResult(
+ $("f3").urlDecode(), "url_decode(f3)", null, STRING().nullable())
+ .testResult(
+ $("f4").urlDecode(),
+ "url_decode(f4)",
+ "inva lid://user:pass@host/file;param?query;p2",
+ STRING())
+ .testResult($("f5").urlDecode(), "url_decode(f5)", "", STRING()),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
+ .onFieldsWithData(
+ "https://flink.apache.org/",
+ "https%3A%2F%2Fflink.apache.org%2F",
+ "http://test?a=b&c=d",
Review Comment:
same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598298446
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
Review Comment:
ok, I will update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598243697
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
we should return the original value if failed, just like Calcite/Spark does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598251586
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ private static TestSetSpec urlDecodeSpec() {
Review Comment:
I think we should be more concise about tests since here it will run flink jobs and it consumes resources and time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2107229466
Thanks for your contribution @superdiaodiao ,
I left some comments
also ci is failed and i tend to think it is related to your changes in python code, could you please have a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2106260978
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606749183
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@superdiaodiao
Do we know why Calcite/Spark do this? It seems to be a strange design choice as we will go forward with this string not decoded, but the subsequent code would expect it to be decoded and is likely to get unexpected results; which would be tricky to debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606797248
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@davidradl
In the Calcite, the author didn't demonstrate the reason and the origin text is: https://github.com/apache/calcite/pull/3318#discussion_r1372603727
While in the Spark, it is different with the Calcite, although the Calcite said that it can be served for Spark. In fact, Spark will throw an exception instead of returning the origin value in this case.
So what's Flink team's expectation? I will follow your guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1603094671
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ @Override
+ Stream<TestSetSpec> getTestSetSpecs() {
+
+ return Stream.of(
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
+ .onFieldsWithData(
+ "https%3A%2F%2Fflink.apache.org%2F",
+ "https://flink.apache.org/",
+ "http%3A%2F%2Ftest%3Fa%3Db%26c%3Dd",
+ null,
+ "inva+lid%3A%2F%2Fuser%3Apass%40host%2Ffile%3Bparam%3Fquery%3Bp2",
+ "")
+ .andDataTypes(STRING(), STRING(), STRING(), STRING(), STRING(), STRING())
+ .testResult(
+ $("f0").urlDecode(),
+ "url_decode(f0)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f1").urlDecode(),
+ "url_decode(f1)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f2").urlDecode(),
+ "url_decode(f2)",
+ "http://test?a=b&c=d",
+ STRING())
+ .testResult(
+ $("f3").urlDecode(), "url_decode(f3)", null, STRING().nullable())
+ .testResult(
+ $("f4").urlDecode(),
+ "url_decode(f4)",
+ "inva lid://user:pass@host/file;param?query;p2",
+ STRING())
+ .testResult($("f5").urlDecode(), "url_decode(f5)", "", STRING()),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
Review Comment:
```suggestion
TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
```
are here tests for `URL_DECODE`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611218326
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
> I would vote for `null` since it will allow to understand, that the function was not able to encode it correctly
You mean, these two funcs should return null when decode or encode failed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2156817722
Thanks for the contribution @superdiaodiao
thanks for the review @HuangXingBo , @davidradl
it looks ok to me, I will test it a bit more and in case of succeed will merge it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611921523
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@snuyanzin @MartijnVisser thankyou very much for your feedback.
Sergey:
Sorry if I was not clear, I did not mean to imply that JAVA runtime Exceptions and errors are the same.
you ask `How are you going to even detect this row with such behavior?` I guess this is not possible for the cast, if we had an error message that included the string that we failed to encode or decode - this could help.
For the null pointer- I was not thinking of a java NullPointerException, just an unexpected null in subsequent processing, maybe if this value ended up being attempted to be written to a non nullable column.
Martjin:
I like your examples of cast and try_cast that was helpful for me. In addition to element (which does not look like a runtime exception) there are also json functions that produce errors (these also do not look like runtime errors).
It seems to me if we can produce a useful error message in the error including the string we could not encode / decode then this could be very helpful to be able to diagnose the issue. If not then returning null looks preferable. Either way I think we should log the runtime error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1603139590
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ @Override
+ Stream<TestSetSpec> getTestSetSpecs() {
+
+ return Stream.of(
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
+ .onFieldsWithData(
+ "https%3A%2F%2Fflink.apache.org%2F",
Review Comment:
3rd row adds with some url parameters, if you think it is useless, I will delete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1623199122
##########
flink-python/pyflink/table/expression.py:
##########
@@ -1273,6 +1273,23 @@ def locate(self, s: Union[str, 'Expression[str]'],
else:
return _ternary_op("locate")(self, s, pos)
+ def url_decode(self) -> 'Expression[str]':
+ """
+ Decodes a given string in 'application/x-www-form-urlencoded' format using the UTF-8
+ encoding scheme. If input is null, or there is an issue with the decoding process(such
+ as encountering an illegal escape pattern), or the encoding scheme is not supported,
+ the function returns null.
+ """
+ return _unary_op("urlDecode")(self)
+
+ def url_encode(self) -> 'Expression[str]':
+ """
+ Translates a string into 'application/x-www-form-urlencoded' format using the UTF-8
+ encoding scheme. If input is null, or there is an issue with the encoding process, or
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611289713
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlEncodeFunction.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_ENCODE}. */
+@Internal
+public class UrlEncodeFunction extends BuiltInScalarFunction {
+
+ public UrlEncodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_ENCODE, context);
+ }
+
+ public @Nullable StringData eval(StringData url) {
+ if (url == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ String value;
+
+ try {
+ value = URLEncoder.encode(url.toString(), charset.name());
+ } catch (UnsupportedEncodingException e) {
+ throw new FlinkRuntimeException(
+ "Failed to encode URL: " + url + " with charset: " + charset.name(), e);
Review Comment:
> Since we don’t plan to support specified Charset, why not call `String encode(String s, Charset charset)` directly, so that we don’t have to catch UnsupportedCharsetException?
In the class URLEncode, it is throwable:
```public static String encode(String s, String enc) throws UnsupportedEncodingException {```
so I think we should catch?
Also, Calcite' code does the same thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2135157310
@snuyanzin @MartijnVisser @HuangXingBo PLZ take a look~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599573380
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ if (value == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new FlinkRuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (FlinkRuntimeException e) {
Review Comment:
yes, however for `catch` only
`throw` should still throw `FlinkRuntimeException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599570079
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ if (value == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new FlinkRuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (FlinkRuntimeException e) {
Review Comment:
You mean FlinkRuntimeException change back to RuntimeException here, like my first version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611771105
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@davidradl thanks for your input
here we are talking about returning null on SQL level so I don't get what NullPointer exactly you mean, could you please elaborate on this?
In Flink there was already a functionality throwing an error (`cast`)
Now the problem with this `cast` is that imagine you have a table with a string column however actually there are numbers written there and we can write a query
```sql
SELECT cast(str_column to INTEGER) from MyTable;
```
now for some reason there is a string `abcd` in one of the records and `cast` is throwing an error on it.
How are you going to even detect this row with such behavior?
with returning `null` it is much easier (`try_cast` could return `null`), we could simply write something like
```sql
SELECT * from (SELECT try_cast(str_column to INTEGER) a, str_column b from MyTable) t WHERE a is NULL;
```
also about
> runtime exceptions like errors (e.g. out of memory).
OutOfMemory is NOT a RuntimeException, it is under Errors
https://docs.oracle.com/javase/8/docs/api/java/lang/OutOfMemoryError.html
https://docs.oracle.com/javase/8/docs/api/java/lang/RuntimeException.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611754275
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
> I would be keener to have an error, it is then obvious this is an out of the ordinary situation
We only do that for `CAST` because CAST explicitly defines that in the SQL standard. That's also why there's a `TRY_CAST`. We don't fail on any of the other functions (except `ELEMENT` if I saw it correctly)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1603093845
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) throws FlinkRuntimeException {
Review Comment:
```suggestion
public @Nullable StringData eval(StringData value) {
```
why do we need this `throws FlinkRuntimeException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1610884892
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
ok, I will make it throwable and add test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2159830832
> hi @snuyanzin @superdiaodiao do we need supports encoding args ?
> db2 https://www.ibm.com/docs/en/db2-for-zos/12?topic=functions-urlencode-urldecode
> max compute https://www.alibabacloud.com/help/en/maxcompute/user-guide/url-decode
>
Calcite, Spark also need only one arg, it is enough to handle cases and UTF-8 can support all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2185697062
As it was mentioned earlier
there is feature freeze currently https://lists.apache.org/thread/ftj0mcs2843yrx1tog9lfhfrbzkrbfvp
so need to wait
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2208669198
> @superdiaodiao can you please rebase to the recent master's version, I think it will help with ci?
OK, I will rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #24773:
URL: https://github.com/apache/flink/pull/24773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611157801
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
I would vote for `null` since it will allow to understand, that the function was not able to encode it correctly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611739983
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
For me, I would be keener to have an error, it is then obvious this is an out of the ordinary situation and as @MartijnVisser says this is going to be extremely unlikely to occur. In this case we would be treating runtime exceptions like errors (e.g. out of memory).
If we return a null then potentially we have some sort of null pointer or lack of integrity in subsequent processing due to the missing data. It would be tricky to understand that this is the root cause.
If the consensus is to return a null,
- would we document that null is returned for runtime errors?
- we should log an error so any unexpected subsequent processing can be diagnosed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611984504
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
How about this:
add a WARN log before returning null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2159808362
hi @snuyanzin @superdiaodiao do we need supports encoding args ?
db2 https://www.ibm.com/docs/en/db2-for-zos/12?topic=functions-urlencode-urldecode
max compute https://www.alibabacloud.com/help/en/maxcompute/user-guide/url-decode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1603142936
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) throws FlinkRuntimeException {
Review Comment:
sorry for this mistake, will be removed...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611984504
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
how about this:
add a WARN log before return null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "HuangXingBo (via GitHub)" <gi...@apache.org>.
HuangXingBo commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611129401
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlEncodeFunction.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_ENCODE}. */
+@Internal
+public class UrlEncodeFunction extends BuiltInScalarFunction {
+
+ public UrlEncodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_ENCODE, context);
+ }
+
+ public @Nullable StringData eval(StringData url) {
+ if (url == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ String value;
+
+ try {
+ value = URLEncoder.encode(url.toString(), charset.name());
+ } catch (UnsupportedEncodingException e) {
+ throw new FlinkRuntimeException(
+ "Failed to encode URL: " + url + " with charset: " + charset.name(), e);
Review Comment:
Since we don’t plan to support specified Charset, why not call `String encode(String s, Charset charset)` directly, so that we don’t have to catch UnsupportedCharsetException?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
Compared with returning the original string, returning null can better confirm whether the parsing failed. The string functions in flink basically follow this specification. However, I have observed the implementation of url_decode implemented by users, and there are two implementations of returning the original string and null, so I have no particular preference for the choice here. cc @lincoln-lil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599510644
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ if (value == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new FlinkRuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (FlinkRuntimeException e) {
Review Comment:
I don't think that here we need to catch `FlinkRuntimeException` since none from this string can throw it
```java
return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
```
`RuntimeException` should be enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599601719
##########
docs/data/sql_functions.yml:
##########
@@ -350,6 +350,17 @@ string:
- sql: LOCATE(string1, string2[, integer])
table: STRING1.locate(STRING2[, INTEGER])
description: Returns the position of the first occurrence of string1 in string2 after position integer. Returns 0 if not found. Returns NULL if any of arguments is NULL.
+ - sql: URL_DECODE(string)
+ table: STRING.urlDecode()
+ description:
+ Decodes a given string in 'application/x-www-form-urlencoded' format using the UTF-8 encoding scheme, will be null if input is null.
+ If there is an issue with the decoding process, such as encountering an illegal escape pattern, the function returns the original input value.
+ If the encoding scheme is not supported, an error will be reported.
Review Comment:
So should I construct a string encoded with UTF_16 to test? Actually I have no idea of this, could you give me some advice? After all, I noticed that the Calcite has no test on non UTF8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2107243083
> Thanks for your contribution @superdiaodiao , I left some comments
>
> also ci is failed and i tend to think it is related to your changes in python code, could you please have a look?
Thanks for your review, it helps a lot.
I will continue to check the python part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2107498923
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598239270
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
Review Comment:
couldn't be `value` null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2157065404
> Thanks for the contribution @superdiaodiao
> thanks for the review @HuangXingBo , @davidradl
>
> it looks ok to me, I will test it a bit more and in case of succeed will merge it
Thank you~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2172705717
Yep, sorry, testing took a bit longer, than expected
however it looks ok
>db2 https://www.ibm.com/docs/en/db2-for-zos/12?topic=functions-urlencode-urldecode
>max compute https://www.alibabacloud.com/help/en/maxcompute/user-guide/url-decode
Thanks for bringing this
I also looked at different vendors:it's hard to find those who supports these functions
And among those who supports it there is no common approach regarding encodings
like StarRocks
https://docs.starrocks.io/docs/3.1/sql-reference/sql-functions/string-functions/url_encode/
Oracle DB itself doesn't support url_encode however Oracle Apex supports it without encoding
So I would go with current implementation as a first step
For different encodings: also need to make a research what encodings should be supported.
And how well it is supported in table module, e.g. I failed to find tests in table module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606776660
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlEncodeFunction.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_ENCODE}. */
+@Internal
+public class UrlEncodeFunction extends BuiltInScalarFunction {
+
+ public UrlEncodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_ENCODE, context);
+ }
+
+ public @Nullable StringData eval(StringData url) {
+ if (url == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ String value;
+
+ try {
+ value = URLEncoder.encode(url.toString(), charset.name());
+ } catch (UnsupportedEncodingException e) {
+ throw new FlinkRuntimeException(
+ "Failed to encode URL: " + url + " with charset: " + charset.name(), e);
Review Comment:
ok, I will add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611365062
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
There aren't many examples of this function in other technologies, but I think most of them don't fail but return `null`. I would also be fine with returning `null` during exceptions. Note: I think in practice it will be quite tricky to get into a failure state with this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1610775137
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
I would follow same approach as Spark does otherwise it's impossible to say whether the function failed or succeeded...
Also we need a test for this case which seems to be absent ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611218326
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
> I would vote for `null` since it will allow to understand, that the function was not able to encode it correctly
@snuyanzin You mean, these two funcs should return null when decode or encode failed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598248764
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
Review Comment:
It could, I will add the null check and return in the start of func.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598262984
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ private static TestSetSpec urlDecodeSpec() {
+ return TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
+ .onFieldsWithData(
+ "https%3A%2F%2Fflink.apache.org%2F",
+ "https://flink.apache.org/",
+ "http%3A%2F%2Ftest%3Fa%3Db%26c%3Dd",
+ null,
+ "aaaa",
+ "inva+lid%3A%2F%2Fuser%3Apass%40host%2Ffile%3Bparam%3Fquery%3Bp2",
+ "")
+ .andDataTypes(STRING(), STRING(), STRING(), STRING(), STRING(), STRING(), STRING())
+ .testResult(
+ $("f0").urlDecode(),
+ "url_decode(f0)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f1").urlEncode().urlDecode(),
+ "url_decode(url_encode(f1))",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f1").urlDecode(),
+ "url_decode(f1)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult($("f2").urlDecode(), "url_decode(f2)", "http://test?a=b&c=d", STRING())
+ .testResult($("f3").urlDecode(), "url_decode(f3)", null, STRING().nullable())
+ .testResult($("f4").urlDecode(), "url_decode(f4)", "aaaa", STRING())
+ .testResult(
+ $("f5").urlDecode(),
+ "url_decode(f5)",
+ "inva lid://user:pass@host/file;param?query;p2",
+ STRING())
+ .testResult($("f6").urlDecode(), "url_decode(f6)", "", STRING());
+ }
+
+ private static TestSetSpec urlEncodeSpec() {
+ return TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
Review Comment:
ok, I will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2107400121
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1617259220
##########
flink-python/pyflink/table/expression.py:
##########
@@ -1273,6 +1273,23 @@ def locate(self, s: Union[str, 'Expression[str]'],
else:
return _ternary_op("locate")(self, s, pos)
+ def url_decode(self) -> 'Expression[str]':
+ """
+ Decodes a given string in 'application/x-www-form-urlencoded' format using the UTF-8
+ encoding scheme. If input is null, or there is an issue with the decoding process(such
Review Comment:
nit: input -> the input
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606797248
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
In the Calcite, the author didn't demonstrate the reason and the origin text is: https://github.com/apache/calcite/pull/3318#discussion_r1372603727
While in the Spark, it is different with the Calcite, although the Calcite said that it can be served for Spark. In fact, Spark will throw an exception instead of returning the origin value in this case.
So what's Flink team's expectation? I will follow your guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611984504
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
How about this:
add a WARN log before returning null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599601719
##########
docs/data/sql_functions.yml:
##########
@@ -350,6 +350,17 @@ string:
- sql: LOCATE(string1, string2[, integer])
table: STRING1.locate(STRING2[, INTEGER])
description: Returns the position of the first occurrence of string1 in string2 after position integer. Returns 0 if not found. Returns NULL if any of arguments is NULL.
+ - sql: URL_DECODE(string)
+ table: STRING.urlDecode()
+ description:
+ Decodes a given string in 'application/x-www-form-urlencoded' format using the UTF-8 encoding scheme, will be null if input is null.
+ If there is an issue with the decoding process, such as encountering an illegal escape pattern, the function returns the original input value.
+ If the encoding scheme is not supported, an error will be reported.
Review Comment:
So should I construct a string encoded with UTF_16 to test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2208233514
@lincoln-lil thanks for having a look
please let us know if there is anything that should be fixed from your point of view
I'm asking since i'm going to merge it at the end of this/beginning of the next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611361668
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
yep, probably it will be better if both return `null`
the problem with failure could be that if we have only one record failing the function it could fail the whole query...
I'm pretty sure this is not the behavior we want... With `null` we seems on a bit safer side
please correct me if i'm wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606749183
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@superdiaodiao
Do we know why Calcite/Spark do this? It seems to be a strange design choice as we will go forward with this string not decoded, but the subsequent code would expect it to be decoded and is likely to get unexpected results; which would be tricky to debug.
I notice that encode does not do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606797248
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@davidradl
In the Calcite, the author didn't demonstrate the reason and the origin text is: https://github.com/apache/calcite/pull/3318#discussion_r1372603727
While in the Spark, it is different from the Calcite, although the Calcite said that it can be served for Spark(which made me think they have the same action at the first time).
In fact, I read its code and tested, Spark will throw an exception instead of returning the origin value in this case.
So what's Flink team's expectation? I will follow your guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606797248
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@davidradl
In the Calcite, the author didn't demonstrate the reason and the origin text is: https://github.com/apache/calcite/pull/3318#discussion_r1372603727
While in the Spark, it is different with the Calcite, although the Calcite said that it can be served for Spark. In fact, I read its code and tested, Spark will throw an exception instead of returning the origin value in this case.
So what's Flink team's expectation? I will follow your guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2106247014
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7b8727596cda8b1feb6edb3c465043ed764fa4c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7b8727596cda8b1feb6edb3c465043ed764fa4c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7b8727596cda8b1feb6edb3c465043ed764fa4c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598262631
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ private static TestSetSpec urlDecodeSpec() {
Review Comment:
ok, make senses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598253206
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ private static TestSetSpec urlDecodeSpec() {
+ return TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
+ .onFieldsWithData(
+ "https%3A%2F%2Fflink.apache.org%2F",
+ "https://flink.apache.org/",
+ "http%3A%2F%2Ftest%3Fa%3Db%26c%3Dd",
+ null,
+ "aaaa",
+ "inva+lid%3A%2F%2Fuser%3Apass%40host%2Ffile%3Bparam%3Fquery%3Bp2",
+ "")
+ .andDataTypes(STRING(), STRING(), STRING(), STRING(), STRING(), STRING(), STRING())
+ .testResult(
+ $("f0").urlDecode(),
+ "url_decode(f0)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f1").urlEncode().urlDecode(),
+ "url_decode(url_encode(f1))",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f1").urlDecode(),
+ "url_decode(f1)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult($("f2").urlDecode(), "url_decode(f2)", "http://test?a=b&c=d", STRING())
+ .testResult($("f3").urlDecode(), "url_decode(f3)", null, STRING().nullable())
+ .testResult($("f4").urlDecode(), "url_decode(f4)", "aaaa", STRING())
+ .testResult(
+ $("f5").urlDecode(),
+ "url_decode(f5)",
+ "inva lid://user:pass@host/file;param?query;p2",
+ STRING())
+ .testResult($("f6").urlDecode(), "url_decode(f6)", "", STRING());
+ }
+
+ private static TestSetSpec urlEncodeSpec() {
+ return TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
Review Comment:
Also here it states that the tests are for `URL_DECODE` however if we count a number of usages of `urlDecode` and `urlEncode` then amount of `urlEncode` is much higher for this test...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599518712
##########
docs/data/sql_functions.yml:
##########
@@ -350,6 +350,17 @@ string:
- sql: LOCATE(string1, string2[, integer])
table: STRING1.locate(STRING2[, INTEGER])
description: Returns the position of the first occurrence of string1 in string2 after position integer. Returns 0 if not found. Returns NULL if any of arguments is NULL.
+ - sql: URL_DECODE(string)
+ table: STRING.urlDecode()
+ description:
+ Decodes a given string in 'application/x-www-form-urlencoded' format using the UTF-8 encoding scheme, will be null if input is null.
+ If there is an issue with the decoding process, such as encountering an illegal escape pattern, the function returns the original input value.
+ If the encoding scheme is not supported, an error will be reported.
Review Comment:
Can we have a test for this?
I guess we need to pass a string like `_UTF16'Hello'` or `_UTF32'Hello'` and may be with usage of some non UTF8 symbols
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599518712
##########
docs/data/sql_functions.yml:
##########
@@ -350,6 +350,17 @@ string:
- sql: LOCATE(string1, string2[, integer])
table: STRING1.locate(STRING2[, INTEGER])
description: Returns the position of the first occurrence of string1 in string2 after position integer. Returns 0 if not found. Returns NULL if any of arguments is NULL.
+ - sql: URL_DECODE(string)
+ table: STRING.urlDecode()
+ description:
+ Decodes a given string in 'application/x-www-form-urlencoded' format using the UTF-8 encoding scheme, will be null if input is null.
+ If there is an issue with the decoding process, such as encountering an illegal escape pattern, the function returns the original input value.
+ If the encoding scheme is not supported, an error will be reported.
Review Comment:
Can we have a test for this?
I guess we need to pass a string like `_UTF16’Hello` or `_UTF32’Hello` and may be with usage of some non UTF8 symbols
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599601719
##########
docs/data/sql_functions.yml:
##########
@@ -350,6 +350,17 @@ string:
- sql: LOCATE(string1, string2[, integer])
table: STRING1.locate(STRING2[, INTEGER])
description: Returns the position of the first occurrence of string1 in string2 after position integer. Returns 0 if not found. Returns NULL if any of arguments is NULL.
+ - sql: URL_DECODE(string)
+ table: STRING.urlDecode()
+ description:
+ Decodes a given string in 'application/x-www-form-urlencoded' format using the UTF-8 encoding scheme, will be null if input is null.
+ If there is an issue with the decoding process, such as encountering an illegal escape pattern, the function returns the original input value.
+ If the encoding scheme is not supported, an error will be reported.
Review Comment:
So should I construct a string encoded with UTF_16 to test? Actually I have no idea of this, could you give me some advice? After all, I noticed that the Calcite has no test on non UTF8.
Maybe we just keep it like Calcite does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598263115
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
Review Comment:
there is no info about such behavior in docs, then it should be updated accordingly imho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1617256004
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ if (value == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException | RuntimeException e) {
+ return null;
Review Comment:
I think we should log the exception here, with the log containing the value.toString(), charset.name() and Exception type and message. This mould mean that the logs could be used to diagnose the cause of the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2185785024
> As it was mentioned earlier there is feature freeze currently https://lists.apache.org/thread/ftj0mcs2843yrx1tog9lfhfrbzkrbfvp so need to wait
OK, get your point. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611415054
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
Yes, returning null makes funcs more stable, 100% agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611921523
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@snuyanzin @MartijnVisser thankyou very much for your feedback.
Sergey:
Sorry if I was not clear, I did not mean to imply that JAVA runtime Exceptions and errors are the same.
you ask `How are you going to even detect this row with such behavior?` I guess this is not possible for the cast, if we had an error message that included the string that we fails to encode or decode - this could help.
For the null pointer- I was not thinking of a java NullPointerException, just an unexpected null in subsequent processing, maybe if this value ended up being attempted to be written to a non nullable column.
Martjin:
I like your examples of cast and try_cast that was helpful for me. In addition to element (which does not look like a runtime exception) there are also json functions that produce errors (these also do not look like runtime errors).
It seems to me if we can produce a useful error message in the error including the string we could not encode / decode then this could be very helpful to be able to diagnose the issue. If not then returning null looks preferable. Either way I think we should log the runtime error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1611921523
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@snuyanzin @MartijnVisser thankyou very much for your feedback.
Sergey:
Sorry if I was not clear, I did not mean to imply that JAVA runtime Exceptions and errors are the same.
you ask `How are you going to even detect this row with such behavior?` I guess this is not possible for the cast, if we had an error message that included the string that we failed to encode or decode - this could help.
For the null pointer- I was not thinking of a java NullPointerException, just an unexpected null in subsequent processing, maybe if this value ended up being attempted to be written to a non nullable column.
Martjin:
I like your examples of cast and try_cast that was helpful for me. In addition to element (which does not look like a runtime exception) there are also json functions that produce errors (these also do not look like runtime errors).
It seems to me if we can produce a useful error message in the error, including the string we could not encode / decode, then this could be very helpful to diagnose the issue. If not then returning null looks preferable. Either way I think we should log the runtime error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1617256860
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlEncodeFunction.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_ENCODE}. */
+@Internal
+public class UrlEncodeFunction extends BuiltInScalarFunction {
+
+ public UrlEncodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_ENCODE, context);
+ }
+
+ public @Nullable StringData eval(StringData url) {
+ if (url == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+
+ try {
+ return StringData.fromString(URLEncoder.encode(url.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ return null;
Review Comment:
I suggest logging the exception here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1617259958
##########
flink-python/pyflink/table/expression.py:
##########
@@ -1273,6 +1273,23 @@ def locate(self, s: Union[str, 'Expression[str]'],
else:
return _ternary_op("locate")(self, s, pos)
+ def url_decode(self) -> 'Expression[str]':
+ """
+ Decodes a given string in 'application/x-www-form-urlencoded' format using the UTF-8
+ encoding scheme. If input is null, or there is an issue with the decoding process(such
+ as encountering an illegal escape pattern), or the encoding scheme is not supported,
+ the function returns null.
+ """
+ return _unary_op("urlDecode")(self)
+
+ def url_encode(self) -> 'Expression[str]':
+ """
+ Translates a string into 'application/x-www-form-urlencoded' format using the UTF-8
+ encoding scheme. If input is null, or there is an issue with the encoding process, or
Review Comment:
nit: input -> the input
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598224511
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
why do we need that?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
Review Comment:
```suggestion
throw new FlinkRuntimeException(
```
We should use `FlinkRuntimeException`
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlEncodeFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_ENCODE}. */
+@Internal
+public class UrlEncodeFunction extends BuiltInScalarFunction {
+
+ public UrlEncodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_ENCODE, context);
+ }
+
+ public @Nullable StringData eval(StringData url) {
+ if (url == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ String value;
+
+ try {
+ value = URLEncoder.encode(url.toString(), charset.name());
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
Review Comment:
```suggestion
throw new FlinkRuntimeException(
```
We should use `FlinkRuntimeException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598236936
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
Review Comment:
OK, I will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1598238010
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlEncodeFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_ENCODE}. */
+@Internal
+public class UrlEncodeFunction extends BuiltInScalarFunction {
+
+ public UrlEncodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_ENCODE, context);
+ }
+
+ public @Nullable StringData eval(StringData url) {
+ if (url == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ String value;
+
+ try {
+ value = URLEncoder.encode(url.toString(), charset.name());
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599507811
##########
docs/data/sql_functions.yml:
##########
@@ -350,6 +350,17 @@ string:
- sql: LOCATE(string1, string2[, integer])
table: STRING1.locate(STRING2[, INTEGER])
description: Returns the position of the first occurrence of string1 in string2 after position integer. Returns 0 if not found. Returns NULL if any of arguments is NULL.
+ - sql: URL_DECODE(string)
Review Comment:
One thing I forgot to mention: we also need to update Chinese version (it could be in English however should be synced)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599570453
##########
docs/data/sql_functions.yml:
##########
@@ -350,6 +350,17 @@ string:
- sql: LOCATE(string1, string2[, integer])
table: STRING1.locate(STRING2[, INTEGER])
description: Returns the position of the first occurrence of string1 in string2 after position integer. Returns 0 if not found. Returns NULL if any of arguments is NULL.
+ - sql: URL_DECODE(string)
Review Comment:
ok, I will add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1599618108
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ if (value == null) {
+ return null;
+ }
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new FlinkRuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (FlinkRuntimeException e) {
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2149090542
@snuyanzin @MartijnVisser @HuangXingBo please take a look~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2172455196
> Thanks for the contribution @superdiaodiao thanks for the review @HuangXingBo , @davidradl
>
> it looks ok to me, I will test it a bit more and in case of succeed will merge it
@snuyanzin Please accept my apologies for the interruption. I was just wondering if there have been any updates on the progress of this PR. I greatly appreciate your help and understanding. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on PR #24773:
URL: https://github.com/apache/flink/pull/24773#issuecomment-2185009821
> Yep, sorry, testing took a bit longer, than expected however it looks ok
......
> UPD: currently there is a feature freeze, let's wait until it will be ended
Hey, may I ask if there have been any updates in the past week? If you need help, just let me know as long as I can provide support. Thanks for your efforts until now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1603144242
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ @Override
+ Stream<TestSetSpec> getTestSetSpecs() {
+
+ return Stream.of(
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
+ .onFieldsWithData(
+ "https%3A%2F%2Fflink.apache.org%2F",
+ "https://flink.apache.org/",
+ "http%3A%2F%2Ftest%3Fa%3Db%26c%3Dd",
+ null,
+ "inva+lid%3A%2F%2Fuser%3Apass%40host%2Ffile%3Bparam%3Fquery%3Bp2",
+ "")
+ .andDataTypes(STRING(), STRING(), STRING(), STRING(), STRING(), STRING())
+ .testResult(
+ $("f0").urlDecode(),
+ "url_decode(f0)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f1").urlDecode(),
+ "url_decode(f1)",
+ "https://flink.apache.org/",
+ STRING())
+ .testResult(
+ $("f2").urlDecode(),
+ "url_decode(f2)",
+ "http://test?a=b&c=d",
+ STRING())
+ .testResult(
+ $("f3").urlDecode(), "url_decode(f3)", null, STRING().nullable())
+ .testResult(
+ $("f4").urlDecode(),
+ "url_decode(f4)",
+ "inva lid://user:pass@host/file;param?query;p2",
+ STRING())
+ .testResult($("f5").urlDecode(), "url_decode(f5)", "", STRING()),
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
Review Comment:
sorry for the typo, will fix right away
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1603091969
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/UrlFunctionsITCase.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+
+public class UrlFunctionsITCase extends BuiltInFunctionTestBase {
+
+ @Override
+ Stream<TestSetSpec> getTestSetSpecs() {
+
+ return Stream.of(
+ TestSetSpec.forFunction(BuiltInFunctionDefinitions.URL_DECODE)
+ .onFieldsWithData(
+ "https%3A%2F%2Fflink.apache.org%2F",
Review Comment:
what is the principal difference between 1st and 3rd rows?
I guess we could remove one of them and reduce amount of flink jobs we are running here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34108][table] Add built-in URL_ENCODE and URL_DECODE function. [flink]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #24773:
URL: https://github.com/apache/flink/pull/24773#discussion_r1606797248
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/UrlDecodeFunction.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.scalar;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.SpecializedFunction;
+
+import javax.annotation.Nullable;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLDecoder;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+/** Implementation of {@link BuiltInFunctionDefinitions#URL_DECODE}. */
+@Internal
+public class UrlDecodeFunction extends BuiltInScalarFunction {
+
+ public UrlDecodeFunction(SpecializedFunction.SpecializedContext context) {
+ super(BuiltInFunctionDefinitions.URL_DECODE, context);
+ }
+
+ public @Nullable StringData eval(StringData value) {
+ final Charset charset = StandardCharsets.UTF_8;
+ try {
+ return StringData.fromString(URLDecoder.decode(value.toString(), charset.name()));
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(
+ "Failed to decode value: " + value + " with charset: " + charset.name(), e);
+ } catch (RuntimeException e) {
+ return value;
+ }
Review Comment:
@davidradl
In the Calcite, the author didn't demonstrate the reason and the origin text is: https://github.com/apache/calcite/pull/3318#discussion_r1372603727
While in the Spark, it is different from the Calcite, although the Calcite said that it can be served for Spark(which made me think they have the same action at the first time). In fact, I read its code and tested, Spark will throw an exception instead of returning the origin value in this case.
So what's Flink team's expectation? I will follow your guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org