You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/08/22 09:29:39 UTC
[GitHub] [inlong] thesumery opened a new pull request, #5627: [INLONG-4764][Sort] Import sort end2end unit test with group file input
thesumery opened a new pull request, #5627:
URL: https://github.com/apache/inlong/pull/5627
### Prepare a Pull Request
*[INLONG-4764][Sort] Import sort end2end unit test with group file input*
*(The following *XYZ* should be replaced by the actual [GitHub Issue](https://github.com/apache/inlong/issues) number)*
- Fixes #4764
### Motivation
*Except sql file test ,we should add group json file test for end2end test.*
### Modifications
*Add group json file test for end2end test*
### Verifying this change
*(Please pick either of the following options)*
- [x] This change is a trivial rework/code cleanup without any test coverage.
- [x] This change added tests and can be verified as follows:
- *Added integration tests for end-to-end test with group json file input*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #5627: [INLONG-4764][Sort] Import sort end2end unit test with group file input
Posted by GitBox <gi...@apache.org>.
EMsnap commented on code in PR #5627:
URL: https://github.com/apache/inlong/pull/5627#discussion_r952082928
##########
inlong-sort/sort-end-to-end-tests/src/test/java/org/apache/inlong/sort/tests/utils/PlaceholderResolver.java:
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sort.tests.utils;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * A file placeholder replacement tool.
+ */
+public class PlaceholderResolver {
+ /**
+ * Default placeholder prefix
+ */
+ public static final String DEFAULT_PLACEHOLDER_PREFIX = "${";
+
+ /**
+ * Default placeholder suffix
+ */
+ public static final String DEFAULT_PLACEHOLDER_SUFFIX = "}";
+
+ /**
+ * Default singleton resolver
+ */
+ private static PlaceholderResolver defaultResolver = new PlaceholderResolver();
+
+ /**
+ * Placeholder prefix
+ */
+ private String placeholderPrefix = DEFAULT_PLACEHOLDER_PREFIX;
+
+ /**
+ * Placeholder suffix
+ */
+ private String placeholderSuffix = DEFAULT_PLACEHOLDER_SUFFIX;
+
+ private PlaceholderResolver() {
+
+ }
+
+ private PlaceholderResolver(String placeholderPrefix, String placeholderSuffix) {
+ this.placeholderPrefix = placeholderPrefix;
+ this.placeholderSuffix = placeholderSuffix;
+ }
+
+ public static PlaceholderResolver getDefaultResolver() {
+ return defaultResolver;
+ }
+
+ public static PlaceholderResolver getResolver(String placeholderPrefix, String placeholderSuffix) {
+ return new PlaceholderResolver(placeholderPrefix, placeholderSuffix);
+ }
+
+ /**
+ * Replace template string with special placeholder according to replace function.
+ * @param content template string with special placeholder
+ * @param rule placeholder replacement rule
+ * @return new replaced string
+ */
+ public String resolveByRule(String content, Function<String, String> rule) {
+ int start = content.indexOf(this.placeholderPrefix);
+ if (start == -1) {
+ return content;
+ }
+ StringBuilder result = new StringBuilder(content);
+ while (start != -1) {
+ int end = result.indexOf(this.placeholderSuffix, start);
+ // get placeholder actual value (e.g. ${id}, get the value represent id)
+ String placeholder = result.substring(start + this.placeholderPrefix.length(), end);
+ // replace placeholder value
+ String replaceContent = placeholder.trim().isEmpty() ? "" : rule.apply(placeholder);
+ result.replace(start, end + this.placeholderSuffix.length(), replaceContent);
+ start = result.indexOf(this.placeholderPrefix, start + replaceContent.length());
+ }
+ return result.toString();
+ }
+
+ /**
+ * Replace template string with special placeholder according to replace function.
+ * @param file template file with special placeholder
+ * @param rule placeholder replacement rule
+ * @return new replaced string
+ */
+ public Path resolveByRule(Path file, Function<String, String> rule) {
+ try {
+ List<String> newContents = Files.readAllLines(file, StandardCharsets.UTF_8)
+ .stream()
+ .map(content -> resolveByRule(content, rule))
+ .collect(Collectors.toList());
+ Path newPath = Paths.get(file.getParent().toString(), file.getFileName() + "$");
+ Files.write(newPath, String.join(System.lineSeparator(), newContents).getBytes(StandardCharsets.UTF_8));
+ return newPath;
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Replace template string with special placeholder according to properties file.
+ * Key is the content of the placeholder <br/><br/>
+ * e.g: content = product:${id}:detail:${did}<br/>
+ * valueMap = id -> 1; pid -> 2<br/>
+ * return: product:1:detail:2<br/>
+ *
+ * @param content template string with special placeholder
+ * @param valueMap placeholder replacement map
+ * @return new replaced string
+ */
+ public String resolveByMap(String content, final Map<String, Object> valueMap) {
+ return resolveByRule(content, placeholderValue -> String.valueOf(valueMap.get(placeholderValue)));
+ }
+
+ /**
+ * Replace template string with special placeholder according to properties file.
+ * Key is the content of the placeholder <br/><br/>
+ * e.g: content = product:${id}:detail:${did}<br/>
+ * valueMap = id -> 1; pid -> 2<br/>
+ * return: product:1:detail:2<br/>
+ *
+ * @param file template string with special placeholder
+ * @param valueMap placeholder replacement map
+ * @return new replaced string
+ */
+ public Path resolveByMap(Path file, final Map<String, Object> valueMap) {
+ return resolveByRule(file, placeholderValue -> String.valueOf(valueMap.get(placeholderValue)));
+ }
+
+ public static void main(String[] args) {
+ String before = "today is ${date}, today weather is ${weather}";
Review Comment:
pls add a test for this rather than a main method
##########
inlong-sort/sort-end-to-end-tests/src/test/resources/env/kafka_test_mysql_init1.txt:
##########
@@ -0,0 +1,19 @@
+CREATE TABLE ${MYSQL_INPUT_TABLE} (
Review Comment:
the file name contains number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #5627: [INLONG-4764][Sort] Import sort end2end unit test with group file input
Posted by GitBox <gi...@apache.org>.
dockerzhang merged PR #5627:
URL: https://github.com/apache/inlong/pull/5627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org