You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "Zkplo (via GitHub)" <gi...@apache.org> on 2024/09/07 06:12:09 UTC
[PR] [INLONG-11018][SDK] Transform SQL supports COMPRESS and UNCOMPRESS Functions [inlong]
Zkplo opened a new pull request, #11048:
URL: https://github.com/apache/inlong/pull/11048
<!-- Prepare a Pull Request
Change the title of pull request refer to the following example:
[INLONG-XYZ][Component] Title of the pull request
-->
<!-- Specify the issue this pull request going to fix.
The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)-->
Fixes #11018
### Motivation
Add two function classes: CompressFunction, UnCompressFunction. Also, add the corresponding unit test classes: TestCompressFunction, TestUnCompressFunction.
### Modifications
<!--Describe the modifications you've done.-->
### Verifying this change
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [x] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11018][SDK] Transform SQL supports COMPRESS and UNCOMPRESS Functions [inlong]
Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang merged PR #11048:
URL: https://github.com/apache/inlong/pull/11048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11018][SDK] Transform SQL supports COMPRESS and UNCOMPRESS Functions [inlong]
Posted by "Zkplo (via GitHub)" <gi...@apache.org>.
Zkplo commented on code in PR #11048:
URL: https://github.com/apache/inlong/pull/11048#discussion_r1749483043
##########
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/CompressFunction.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Function;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.Deflater;
+
+/**
+ * CompressFunction
+ * description: compress(string_to_compress) : Compresses a string
+ * - Return NULL if string_to_compress is NULL.
+ * - Return "" if string_to_compress is "".
+ * - Return the result as a binary string.
+ */
+@Slf4j
+@TransformFunction(names = {"compress"})
+public class CompressFunction implements ValueParser {
+
+ private final ValueParser stringParser;
+ // ISO-8859-1 encoding does not convert negative numbers and can preserve the original values of byte arrays.
+ private final Charset CHARSET = StandardCharsets.ISO_8859_1;
+
+ public CompressFunction(Function expr) {
+ stringParser = OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
Review Comment:
As stated on the MySQL official website（ https://dev.mysql.com/doc/refman/8.4/en/encryption-functions.html#function_compress ）, Compress Function uses zlib for string compression, and only needs to provide the string to be compressed as the parameter of the function. The CHARSET here is used to perform lossless conversion between the compressed byte array and string to meet the output requirements of Compress().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11018][SDK] Transform SQL supports COMPRESS and UNCOMPRESS Functions [inlong]
Posted by "Zkplo (via GitHub)" <gi...@apache.org>.
Zkplo commented on code in PR #11048:
URL: https://github.com/apache/inlong/pull/11048#discussion_r1751263395
##########
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/CompressFunction.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Function;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.Deflater;
+
+/**
+ * CompressFunction
+ * description: compress(string_to_compress) : Compresses a string
+ * - Return NULL if string_to_compress is NULL.
+ * - Return "" if string_to_compress is "".
+ * - Return the result as a binary string.
+ */
+@Slf4j
+@TransformFunction(names = {"compress"})
+public class CompressFunction implements ValueParser {
+
+ private final ValueParser stringParser;
+ // ISO-8859-1 encoding does not convert negative numbers and can preserve the original values of byte arrays.
+ private final Charset CHARSET = StandardCharsets.ISO_8859_1;
+
+ public CompressFunction(Function expr) {
+ stringParser = OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
Review Comment:
done.
##########
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/UnCompressFunction.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Function;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.zip.Inflater;
+
+/**
+ * UnCompressFunction
+ * description: uncompress(string_to_uncompress) :
+ * - Return NULL if string_to_uncompress is NULL.
+ * - Return "" if string_to_uncompress is "".
+ * - Return the result as a string.
+ */
+@Slf4j
+@TransformFunction(names = {"uncompress"})
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11018][SDK] Transform SQL supports COMPRESS and UNCOMPRESS Functions [inlong]
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on code in PR #11048:
URL: https://github.com/apache/inlong/pull/11048#discussion_r1750430979
##########
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/CompressFunction.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Function;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.Deflater;
+
+/**
+ * CompressFunction
+ * description: compress(string_to_compress) : Compresses a string
+ * - Return NULL if string_to_compress is NULL.
+ * - Return "" if string_to_compress is "".
+ * - Return the result as a binary string.
+ */
+@Slf4j
+@TransformFunction(names = {"compress"})
+public class CompressFunction implements ValueParser {
+
+ private final ValueParser stringParser;
+ // ISO-8859-1 encoding does not convert negative numbers and can preserve the original values of byte arrays.
+ private final Charset CHARSET = StandardCharsets.ISO_8859_1;
+
+ public CompressFunction(Function expr) {
+ stringParser = OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
Review Comment:
provide a two arguments method like compress(string_to_compress, compress_type)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11018][SDK] Transform SQL supports COMPRESS and UNCOMPRESS Functions [inlong]
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on code in PR #11048:
URL: https://github.com/apache/inlong/pull/11048#discussion_r1750430979
##########
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/CompressFunction.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Function;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.Deflater;
+
+/**
+ * CompressFunction
+ * description: compress(string_to_compress) : Compresses a string
+ * - Return NULL if string_to_compress is NULL.
+ * - Return "" if string_to_compress is "".
+ * - Return the result as a binary string.
+ */
+@Slf4j
+@TransformFunction(names = {"compress"})
+public class CompressFunction implements ValueParser {
+
+ private final ValueParser stringParser;
+ // ISO-8859-1 encoding does not convert negative numbers and can preserve the original values of byte arrays.
+ private final Charset CHARSET = StandardCharsets.ISO_8859_1;
+
+ public CompressFunction(Function expr) {
+ stringParser = OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
Review Comment:
provide a two arguments function like compress(string_to_compress, compress_type)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11018][SDK] Transform SQL supports COMPRESS and UNCOMPRESS Functions [inlong]
Posted by "vernedeng (via GitHub)" <gi...@apache.org>.
vernedeng commented on code in PR #11048:
URL: https://github.com/apache/inlong/pull/11048#discussion_r1750431867
##########
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/UnCompressFunction.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Function;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.zip.Inflater;
+
+/**
+ * UnCompressFunction
+ * description: uncompress(string_to_uncompress) :
+ * - Return NULL if string_to_uncompress is NULL.
+ * - Return "" if string_to_uncompress is "".
+ * - Return the result as a string.
+ */
+@Slf4j
+@TransformFunction(names = {"uncompress"})
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-11018][SDK] Transform SQL supports COMPRESS and UNCOMPRESS Functions [inlong]
Posted by "luchunliang (via GitHub)" <gi...@apache.org>.
luchunliang commented on code in PR #11048:
URL: https://github.com/apache/inlong/pull/11048#discussion_r1749438886
##########
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/function/CompressFunction.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.transform.process.function;
+
+import org.apache.inlong.sdk.transform.decode.SourceData;
+import org.apache.inlong.sdk.transform.process.Context;
+import org.apache.inlong.sdk.transform.process.operator.OperatorTools;
+import org.apache.inlong.sdk.transform.process.parser.ValueParser;
+
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Function;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.Deflater;
+
+/**
+ * CompressFunction
+ * description: compress(string_to_compress) : Compresses a string
+ * - Return NULL if string_to_compress is NULL.
+ * - Return "" if string_to_compress is "".
+ * - Return the result as a binary string.
+ */
+@Slf4j
+@TransformFunction(names = {"compress"})
+public class CompressFunction implements ValueParser {
+
+ private final ValueParser stringParser;
+ // ISO-8859-1 encoding does not convert negative numbers and can preserve the original values of byte arrays.
+ private final Charset CHARSET = StandardCharsets.ISO_8859_1;
+
+ public CompressFunction(Function expr) {
+ stringParser = OperatorTools.buildParser(expr.getParameters().getExpressions().get(0));
Review Comment:
Miss the parameter of compress type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org