You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "SbloodyS (via GitHub)" <gi...@apache.org> on 2024/09/13 03:00:39 UTC
[PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
SbloodyS opened a new pull request, #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613
<!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! -->
## Purpose of the pull request
close #16096
close #16097
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
(or)
## Pull Request Notice
[Pull Request Notice](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)
If your pull request contain incompatible change, you should also add it to `docs/docs/en/guide/upgrede/incompatible.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#issuecomment-2366262279
Please retry analysis of this Pull-Request directly on SonarCloud
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#discussion_r1758126001
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/config/IPropertyDelegate.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.config;
+
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Function;
+
+public interface IPropertyDelegate {
+
+ String get(String key);
+
+ String get(String key, String defaultValue);
+
+ Set<String> getPropertyKeys();
+
+ default Optional<String> getOptional(String key) {
+ return getOptional(key, Function.identity());
+ }
+
+ default Integer getInt(String key) {
+ return get(key, Integer::parseInt);
+ }
+
+ default Integer getInt(String key, Integer defaultValue) {
+ return get(key, Integer::parseInt, defaultValue);
Review Comment:
## Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4861)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/config/IPropertyDelegate.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.config;
+
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Function;
+
+public interface IPropertyDelegate {
+
+ String get(String key);
+
+ String get(String key, String defaultValue);
+
+ Set<String> getPropertyKeys();
+
+ default Optional<String> getOptional(String key) {
+ return getOptional(key, Function.identity());
+ }
+
+ default Integer getInt(String key) {
+ return get(key, Integer::parseInt);
Review Comment:
## Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4860)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/JSONUtils.java:
##########
@@ -0,0 +1,427 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.utils;
+
+import static com.fasterxml.jackson.databind.DeserializationFeature.ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT;
+import static com.fasterxml.jackson.databind.DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES;
+import static com.fasterxml.jackson.databind.DeserializationFeature.READ_UNKNOWN_ENUM_VALUES_AS_NULL;
+import static com.fasterxml.jackson.databind.MapperFeature.REQUIRE_SETTERS_FOR_GETTERS;
+import static com.fasterxml.jackson.databind.SerializationFeature.FAIL_ON_EMPTY_BEANS;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.dolphinscheduler.spi.constants.DateConstants.YYYY_MM_DD_HH_MM_SS;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TimeZone;
+
+import javax.annotation.Nullable;
+
+import lombok.extern.slf4j.Slf4j;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectWriter;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.json.JsonMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fasterxml.jackson.databind.node.TextNode;
+import com.fasterxml.jackson.databind.type.CollectionType;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import com.google.common.base.Strings;
+
+@Slf4j
+public final class JSONUtils {
+
+ static {
+ log.info("init timezone: {}", TimeZone.getDefault());
+ }
+
+ private static final ObjectMapper objectMapper = JsonMapper.builder()
+ .configure(FAIL_ON_UNKNOWN_PROPERTIES, false)
+ .configure(ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT, true)
+ .configure(READ_UNKNOWN_ENUM_VALUES_AS_NULL, true)
+ .configure(REQUIRE_SETTERS_FOR_GETTERS, true)
+ .configure(FAIL_ON_EMPTY_BEANS, false)
+ .addModule(new JavaTimeModule())
+ .addModule(new SimpleModule()
+ .addSerializer(LocalDateTime.class, new LocalDateTimeSerializer())
+ .addDeserializer(LocalDateTime.class, new LocalDateTimeDeserializer()))
+ .defaultTimeZone(TimeZone.getDefault())
+ .defaultDateFormat(new SimpleDateFormat(YYYY_MM_DD_HH_MM_SS))
+ .build();
+
+ private JSONUtils() {
+ throw new UnsupportedOperationException("Construct JSONUtils");
+ }
+
+ public static synchronized void setTimeZone(TimeZone timeZone) {
+ objectMapper.setTimeZone(timeZone);
+ }
+
+ public static ArrayNode createArrayNode() {
+ return objectMapper.createArrayNode();
+ }
+
+ public static ObjectNode createObjectNode() {
+ return objectMapper.createObjectNode();
+ }
+
+ public static JsonNode toJsonNode(Object obj) {
+ return objectMapper.valueToTree(obj);
+ }
+
+ /**
+ * json representation of object
+ *
+ * @param object object
+ * @param feature feature
+ * @return object to json string
+ */
+ public static String toJsonString(Object object, SerializationFeature feature) {
+ try {
+ ObjectWriter writer = objectMapper.writer(feature);
+ return writer.writeValueAsString(object);
+ } catch (Exception e) {
+ log.error("object to json exception!", e);
+ }
+
+ return null;
+ }
+
+ /**
+ * This method deserializes the specified Json into an object of the specified class. It is not
+ * suitable to use if the specified class is a generic type since it will not have the generic
+ * type information because of the Type Erasure feature of Java. Therefore, this method should not
+ * be used if the desired type is a generic type. Note that this method works fine if the any of
+ * the fields of the specified object are generics, just the object itself should not be a
+ * generic type.
+ *
+ * @param json the string from which the object is to be deserialized
+ * @param clazz the class of T
+ * @param <T> T
+ * @return an object of type T from the string
+ * classOfT
+ */
+ public static @Nullable <T> T parseObject(String json, Class<T> clazz) {
+ if (Strings.isNullOrEmpty(json)) {
+ return null;
+ }
+
+ try {
+ return objectMapper.readValue(json, clazz);
+ } catch (Exception e) {
+ log.error("Parse object exception, jsonStr: {}, class: {}", json, clazz, e);
+ }
+ return null;
+ }
+
+ /**
+ * deserialize
+ *
+ * @param src byte array
+ * @param clazz class
+ * @param <T> deserialize type
+ * @return deserialize type
+ */
+ public static <T> T parseObject(byte[] src, Class<T> clazz) {
+ if (src == null) {
+ return null;
+ }
+ String json = new String(src, UTF_8);
+ return parseObject(json, clazz);
+ }
+
+ /**
+ * json to list
+ *
+ * @param json json string
+ * @param clazz class
+ * @param <T> T
+ * @return list
+ */
+ public static <T> List<T> toList(String json, Class<T> clazz) {
+ if (Strings.isNullOrEmpty(json)) {
+ return Collections.emptyList();
+ }
+
+ try {
+ CollectionType listType = objectMapper.getTypeFactory().constructCollectionType(ArrayList.class, clazz);
+ return objectMapper.readValue(json, listType);
+ } catch (Exception e) {
+ log.error("parse list exception!", e);
+ }
+
+ return Collections.emptyList();
+ }
+
+ /**
+ * check json object valid
+ *
+ * @param json json
+ * @return true if valid
+ */
+ public static boolean checkJsonValid(String json) {
+ return checkJsonValid(json, true);
+ }
+
+ public static boolean checkJsonValid(String json, Boolean logFlag) {
+ if (Strings.isNullOrEmpty(json)) {
+ return false;
+ }
+
+ try {
+ objectMapper.readTree(json);
+ return true;
+ } catch (IOException e) {
+ if (logFlag)
+ log.error("check json object valid exception!", e);
+ }
+
+ return false;
+ }
+
+ /**
+ * Method for finding a JSON Object field with specified name in this
+ * node or its child nodes, and returning value it has.
+ * If no matching field is found in this node or its descendants, returns null.
+ *
+ * @param jsonNode json node
+ * @param fieldName Name of field to look for
+ * @return Value of first matching node found, if any; null if none
+ */
+ public static String findValue(JsonNode jsonNode, String fieldName) {
+ JsonNode node = jsonNode.findValue(fieldName);
+
+ if (node == null) {
+ return null;
+ }
+
+ return node.asText();
+ }
+
+ /**
+ * json to map
+ * {@link #toMap(String, Class, Class)}
+ *
+ * @param json json
+ * @return json to map
+ */
+ public static Map<String, String> toMap(String json) {
+ return parseObject(json, new TypeReference<Map<String, String>>() {
+ });
+ }
+
+ /**
+ * json to map
+ *
+ * @param json json
+ * @param classK classK
+ * @param classV classV
+ * @param <K> K
+ * @param <V> V
+ * @return to map
+ */
+ public static <K, V> Map<K, V> toMap(String json, Class<K> classK, Class<V> classV) {
Review Comment:
## Useless parameter
The parameter 'classK' is never used.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4870)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/shell/AbstractShell.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.shell;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * A base class for running a Unix command.
+ *
+ * <code>AbstractShell</code> can be used to run unix commands like <code>du</code> or
+ * <code>df</code>. It also offers facilities to gate commands by
+ * time-intervals.
+ */
+@Slf4j
+public abstract class AbstractShell {
+
+ /**
+ * Time after which the executing script would be timedout
+ */
+ protected long timeOutInterval = 0L;
+ /**
+ * If or not script timed out
+ */
+ private AtomicBoolean timedOut;
+
+ /**
+ * refresh interval in msec
+ */
+ private long interval;
+
+ /**
+ * last time the command was performed
+ */
+ private long lastTime;
+
+ /**
+ * env for the command execution
+ */
+ private Map<String, String> environment;
+ private File dir;
+
+ /**
+ * sub process used to execute the command
+ */
+ private Process process;
+ private int exitCode;
+
+ /**
+ * If or not script finished executing
+ */
+ private AtomicBoolean completed;
+
+ public AbstractShell() {
+ this(0L);
+ }
+
+ /**
+ * @param interval the minimum duration to wait before re-executing the
+ * command.
+ */
+ public AbstractShell(long interval) {
+ this.interval = interval;
+ this.lastTime = (interval < 0) ? 0 : -interval;
+ }
+
+ /**
+ * set the environment for the command
+ * @param env Mapping of environment variables
+ */
+ protected void setEnvironment(Map<String, String> env) {
+ this.environment = env;
+ }
+
+ /**
+ * set the working directory
+ * @param dir The directory where the command would be executed
+ */
+ protected void setWorkingDirectory(File dir) {
+ this.dir = dir;
+ }
+
+ /**
+ * check to see if a command needs to be executed and execute if needed
+ * @throws IOException errors
+ */
+ protected void run() throws IOException {
+ if (lastTime + interval > System.currentTimeMillis()) {
+ return;
+ }
+ // reset for next run
+ exitCode = 0;
+ runCommand();
+ }
+
+ /**
+ * Run a command actual work
+ */
+ private void runCommand() throws IOException {
+ ProcessBuilder builder = new ProcessBuilder(getExecString());
+ Timer timeOutTimer = null;
+ ShellTimeoutTimerTask timeoutTimerTask = null;
+ timedOut = new AtomicBoolean(false);
+ completed = new AtomicBoolean(false);
+
+ if (environment != null) {
+ builder.environment().putAll(this.environment);
+ }
+ if (dir != null) {
+ builder.directory(this.dir);
+ }
+
+ process = builder.start();
+ ProcessContainer.putProcess(process);
+
+ if (timeOutInterval > 0) {
+ timeOutTimer = new Timer();
+ timeoutTimerTask = new ShellTimeoutTimerTask(this);
+ // One time scheduling.
+ timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);
+ }
+ final BufferedReader errReader =
+ new BufferedReader(
+ new InputStreamReader(process.getErrorStream()));
+ BufferedReader inReader =
+ new BufferedReader(
+ new InputStreamReader(process.getInputStream()));
+ final StringBuilder errMsg = new StringBuilder();
+
+ // read error and input streams as this would free up the buffers
+ // free the error stream buffer
+ Thread errThread = new Thread() {
+
+ @Override
+ public void run() {
+ try {
+ String line = errReader.readLine();
+ while ((line != null) && !isInterrupted()) {
+ errMsg.append(line);
+ errMsg.append(System.getProperty("line.separator"));
+ line = errReader.readLine();
+ }
+ } catch (IOException ioe) {
+ log.warn("Error reading the error stream", ioe);
+ }
+ }
+ };
+ Thread inThread = new Thread() {
+
+ @Override
+ public void run() {
+ try {
+ parseExecResult(inReader);
+ } catch (IOException ioe) {
+ log.warn("Error reading the in stream", ioe);
+ }
+ super.run();
+ }
+ };
+ try {
+ errThread.start();
+ inThread.start();
+ } catch (IllegalStateException ise) {
+ log.warn("Illegal while starting the error and in thread", ise);
+ }
+ try {
+ // parse the output
+ exitCode = process.waitFor();
+ try {
+ // make sure that the error and in thread exits
+ errThread.join();
+ inThread.join();
+ } catch (InterruptedException ie) {
+ log.warn("Interrupted while reading the error and in stream", ie);
+ }
+ completed.compareAndSet(false, true);
+ // the timeout thread handling
+ // taken care in finally block
+ if (exitCode != 0 || errMsg.length() > 0) {
+ throw new ExitCodeException(exitCode, errMsg.toString());
+ }
+ } catch (InterruptedException ie) {
+ throw new IOException(ie.toString());
+ } finally {
+ if ((timeOutTimer != null) && !timedOut.get()) {
+ timeOutTimer.cancel();
+ }
+ // close the input stream
+ try {
+ inReader.close();
+ } catch (IOException ioe) {
+ log.warn("Error while closing the input stream", ioe);
+ }
+ if (!completed.get()) {
+ errThread.interrupt();
+ }
+ try {
+ errReader.close();
+ } catch (IOException ioe) {
+ log.warn("Error while closing the error stream", ioe);
+ }
+ ProcessContainer.removeProcess(process);
+ process.destroy();
+ lastTime = System.currentTimeMillis();
+ }
+ }
+
+ /**
+ *
+ * @return an array containing the command name and its parameters
+ */
+ protected abstract String[] getExecString();
+
+ /**
+ * Parse the execution result
+ * @param lines lines
+ * @throws IOException errors
+ */
+ protected abstract void parseExecResult(BufferedReader lines) throws IOException;
+
+ /**
+ * get the current sub-process executing the given command
+ * @return process executing the command
+ */
+ public Process getProcess() {
+ return process;
+ }
+
+ /** get the exit code
+ * @return the exit code of the process
+ */
+ public int getExitCode() {
+ return exitCode;
+ }
+
+ /**
+ * Set if the command has timed out.
+ *
+ */
+ private void setTimedOut() {
+ this.timedOut.set(true);
+ }
+
+ /**
+ * Timer which is used to timeout scripts spawned off by shell.
+ */
+ private static class ShellTimeoutTimerTask extends TimerTask {
+
+ private AbstractShell shell;
+
+ public ShellTimeoutTimerTask(AbstractShell shell) {
+ this.shell = shell;
+ }
+
+ @Override
+ public void run() {
+ Process p = shell.getProcess();
+ try {
+ p.exitValue();
Review Comment:
## Dereferenced variable may be null
Variable [p](1) may be null at this access as suggested by [this](2) null guard.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4872)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/config/IPropertyDelegate.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.config;
+
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Function;
+
+public interface IPropertyDelegate {
+
+ String get(String key);
+
+ String get(String key, String defaultValue);
+
+ Set<String> getPropertyKeys();
+
+ default Optional<String> getOptional(String key) {
+ return getOptional(key, Function.identity());
+ }
+
+ default Integer getInt(String key) {
+ return get(key, Integer::parseInt);
+ }
+
+ default Integer getInt(String key, Integer defaultValue) {
+ return get(key, Integer::parseInt, defaultValue);
+ }
+
+ default Long getLong(String key) {
+ return get(key, Long::parseLong);
Review Comment:
## Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4862)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/FileUtils.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.utils;
+
+import static org.apache.dolphinscheduler.spi.constants.Constants.FORMAT_S_S;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.SystemUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.FileAlreadyExistsException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Set;
+
+import lombok.NonNull;
+import lombok.experimental.UtilityClass;
+import lombok.extern.slf4j.Slf4j;
+
+@UtilityClass
+@Slf4j
+public class FileUtils {
+
+ public static final String KUBE_CONFIG_FILE = "config";
+
+ private static final Set<PosixFilePermission> PERMISSION_755 = PosixFilePermissions.fromString("rwxr-xr-x");
+
+ /**
+ * absolute path of kubernetes configuration file
+ *
+ * @param execPath
+ * @return
+ */
+ public static String getKubeConfigPath(String execPath) {
+ return String.format(FORMAT_S_S, execPath, KUBE_CONFIG_FILE);
+ }
+
+ public static void createFileWith755(@NonNull Path path) throws IOException {
+ if (SystemUtils.IS_OS_WINDOWS) {
+ Files.createFile(path);
+ } else {
+ Files.createFile(path);
+ Files.setPosixFilePermissions(path, PERMISSION_755);
+ }
+ }
+
+ public static void createDirectoryWith755(@NonNull Path path) throws IOException {
+ if (path.toFile().exists()) {
Review Comment:
## Uncontrolled data used in path expression
This path depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4866)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/FileUtils.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.utils;
+
+import static org.apache.dolphinscheduler.spi.constants.Constants.FORMAT_S_S;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.SystemUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.FileAlreadyExistsException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Set;
+
+import lombok.NonNull;
+import lombok.experimental.UtilityClass;
+import lombok.extern.slf4j.Slf4j;
+
+@UtilityClass
+@Slf4j
+public class FileUtils {
+
+ public static final String KUBE_CONFIG_FILE = "config";
+
+ private static final Set<PosixFilePermission> PERMISSION_755 = PosixFilePermissions.fromString("rwxr-xr-x");
+
+ /**
+ * absolute path of kubernetes configuration file
+ *
+ * @param execPath
+ * @return
+ */
+ public static String getKubeConfigPath(String execPath) {
+ return String.format(FORMAT_S_S, execPath, KUBE_CONFIG_FILE);
+ }
+
+ public static void createFileWith755(@NonNull Path path) throws IOException {
+ if (SystemUtils.IS_OS_WINDOWS) {
+ Files.createFile(path);
+ } else {
+ Files.createFile(path);
+ Files.setPosixFilePermissions(path, PERMISSION_755);
+ }
+ }
+
+ public static void createDirectoryWith755(@NonNull Path path) throws IOException {
+ if (path.toFile().exists()) {
+ return;
+ }
+ if (OSUtils.isWindows()) {
+ Files.createDirectories(path);
Review Comment:
## Uncontrolled data used in path expression
This path depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4867)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/config/ImmutablePropertyDelegate.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.config;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Properties;
+import java.util.Set;
+
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * This class is used to get the properties from the classpath.
+ */
+@Slf4j
+public class ImmutablePropertyDelegate implements IPropertyDelegate {
+
+ private static final String COMMON_PROPERTIES_NAME = "/common.properties";
+
+ private final Properties properties;
+
+ public ImmutablePropertyDelegate() {
+ this(COMMON_PROPERTIES_NAME);
+ }
+
+ public ImmutablePropertyDelegate(String... propertyAbsolutePath) {
+ properties = new Properties();
+ // read from classpath
+ for (String fileName : propertyAbsolutePath) {
+ try (InputStream fis = getClass().getResourceAsStream(fileName)) {
Review Comment:
## Unsafe use of getResource
The idiom getClass().getResource() is unsafe for classes that may be extended.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4873)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/config/IPropertyDelegate.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.config;
+
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Function;
+
+public interface IPropertyDelegate {
+
+ String get(String key);
+
+ String get(String key, String defaultValue);
+
+ Set<String> getPropertyKeys();
+
+ default Optional<String> getOptional(String key) {
+ return getOptional(key, Function.identity());
+ }
+
+ default Integer getInt(String key) {
+ return get(key, Integer::parseInt);
+ }
+
+ default Integer getInt(String key, Integer defaultValue) {
+ return get(key, Integer::parseInt, defaultValue);
+ }
+
+ default Long getLong(String key) {
+ return get(key, Long::parseLong);
+ }
+
+ default Long getLong(String key, Long defaultValue) {
+ return get(key, Long::parseLong, defaultValue);
+ }
+
+ default Double getDouble(String key) {
+ return get(key, Double::parseDouble);
+ }
+
+ default Double getDouble(String key, Double defaultValue) {
+ return get(key, Double::parseDouble, defaultValue);
Review Comment:
## Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4865)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/FileUtils.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.utils;
+
+import static org.apache.dolphinscheduler.spi.constants.Constants.FORMAT_S_S;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.SystemUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.FileAlreadyExistsException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Set;
+
+import lombok.NonNull;
+import lombok.experimental.UtilityClass;
+import lombok.extern.slf4j.Slf4j;
+
+@UtilityClass
+@Slf4j
+public class FileUtils {
+
+ public static final String KUBE_CONFIG_FILE = "config";
+
+ private static final Set<PosixFilePermission> PERMISSION_755 = PosixFilePermissions.fromString("rwxr-xr-x");
+
+ /**
+ * absolute path of kubernetes configuration file
+ *
+ * @param execPath
+ * @return
+ */
+ public static String getKubeConfigPath(String execPath) {
+ return String.format(FORMAT_S_S, execPath, KUBE_CONFIG_FILE);
+ }
+
+ public static void createFileWith755(@NonNull Path path) throws IOException {
+ if (SystemUtils.IS_OS_WINDOWS) {
+ Files.createFile(path);
+ } else {
+ Files.createFile(path);
+ Files.setPosixFilePermissions(path, PERMISSION_755);
+ }
+ }
+
+ public static void createDirectoryWith755(@NonNull Path path) throws IOException {
+ if (path.toFile().exists()) {
+ return;
+ }
+ if (OSUtils.isWindows()) {
+ Files.createDirectories(path);
+ } else {
+ Path parent = path.getParent();
+ if (parent != null && !parent.toFile().exists()) {
Review Comment:
## Uncontrolled data used in path expression
This path depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4868)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/config/IPropertyDelegate.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.config;
+
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Function;
+
+public interface IPropertyDelegate {
+
+ String get(String key);
+
+ String get(String key, String defaultValue);
+
+ Set<String> getPropertyKeys();
+
+ default Optional<String> getOptional(String key) {
+ return getOptional(key, Function.identity());
+ }
+
+ default Integer getInt(String key) {
+ return get(key, Integer::parseInt);
+ }
+
+ default Integer getInt(String key, Integer defaultValue) {
+ return get(key, Integer::parseInt, defaultValue);
+ }
+
+ default Long getLong(String key) {
+ return get(key, Long::parseLong);
+ }
+
+ default Long getLong(String key, Long defaultValue) {
+ return get(key, Long::parseLong, defaultValue);
Review Comment:
## Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4863)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/config/IPropertyDelegate.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.config;
+
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Function;
+
+public interface IPropertyDelegate {
+
+ String get(String key);
+
+ String get(String key, String defaultValue);
+
+ Set<String> getPropertyKeys();
+
+ default Optional<String> getOptional(String key) {
+ return getOptional(key, Function.identity());
+ }
+
+ default Integer getInt(String key) {
+ return get(key, Integer::parseInt);
+ }
+
+ default Integer getInt(String key, Integer defaultValue) {
+ return get(key, Integer::parseInt, defaultValue);
+ }
+
+ default Long getLong(String key) {
+ return get(key, Long::parseLong);
+ }
+
+ default Long getLong(String key, Long defaultValue) {
+ return get(key, Long::parseLong, defaultValue);
+ }
+
+ default Double getDouble(String key) {
+ return get(key, Double::parseDouble);
Review Comment:
## Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4864)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/JSONUtils.java:
##########
@@ -0,0 +1,427 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.utils;
+
+import static com.fasterxml.jackson.databind.DeserializationFeature.ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT;
+import static com.fasterxml.jackson.databind.DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES;
+import static com.fasterxml.jackson.databind.DeserializationFeature.READ_UNKNOWN_ENUM_VALUES_AS_NULL;
+import static com.fasterxml.jackson.databind.MapperFeature.REQUIRE_SETTERS_FOR_GETTERS;
+import static com.fasterxml.jackson.databind.SerializationFeature.FAIL_ON_EMPTY_BEANS;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.dolphinscheduler.spi.constants.DateConstants.YYYY_MM_DD_HH_MM_SS;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TimeZone;
+
+import javax.annotation.Nullable;
+
+import lombok.extern.slf4j.Slf4j;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectWriter;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.json.JsonMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fasterxml.jackson.databind.node.TextNode;
+import com.fasterxml.jackson.databind.type.CollectionType;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import com.google.common.base.Strings;
+
+@Slf4j
+public final class JSONUtils {
+
+ static {
+ log.info("init timezone: {}", TimeZone.getDefault());
+ }
+
+ private static final ObjectMapper objectMapper = JsonMapper.builder()
+ .configure(FAIL_ON_UNKNOWN_PROPERTIES, false)
+ .configure(ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT, true)
+ .configure(READ_UNKNOWN_ENUM_VALUES_AS_NULL, true)
+ .configure(REQUIRE_SETTERS_FOR_GETTERS, true)
+ .configure(FAIL_ON_EMPTY_BEANS, false)
+ .addModule(new JavaTimeModule())
+ .addModule(new SimpleModule()
+ .addSerializer(LocalDateTime.class, new LocalDateTimeSerializer())
+ .addDeserializer(LocalDateTime.class, new LocalDateTimeDeserializer()))
+ .defaultTimeZone(TimeZone.getDefault())
+ .defaultDateFormat(new SimpleDateFormat(YYYY_MM_DD_HH_MM_SS))
+ .build();
+
+ private JSONUtils() {
+ throw new UnsupportedOperationException("Construct JSONUtils");
+ }
+
+ public static synchronized void setTimeZone(TimeZone timeZone) {
+ objectMapper.setTimeZone(timeZone);
+ }
+
+ public static ArrayNode createArrayNode() {
+ return objectMapper.createArrayNode();
+ }
+
+ public static ObjectNode createObjectNode() {
+ return objectMapper.createObjectNode();
+ }
+
+ public static JsonNode toJsonNode(Object obj) {
+ return objectMapper.valueToTree(obj);
+ }
+
+ /**
+ * json representation of object
+ *
+ * @param object object
+ * @param feature feature
+ * @return object to json string
+ */
+ public static String toJsonString(Object object, SerializationFeature feature) {
+ try {
+ ObjectWriter writer = objectMapper.writer(feature);
+ return writer.writeValueAsString(object);
+ } catch (Exception e) {
+ log.error("object to json exception!", e);
+ }
+
+ return null;
+ }
+
+ /**
+ * This method deserializes the specified Json into an object of the specified class. It is not
+ * suitable to use if the specified class is a generic type since it will not have the generic
+ * type information because of the Type Erasure feature of Java. Therefore, this method should not
+ * be used if the desired type is a generic type. Note that this method works fine if the any of
+ * the fields of the specified object are generics, just the object itself should not be a
+ * generic type.
+ *
+ * @param json the string from which the object is to be deserialized
+ * @param clazz the class of T
+ * @param <T> T
+ * @return an object of type T from the string
+ * classOfT
+ */
+ public static @Nullable <T> T parseObject(String json, Class<T> clazz) {
+ if (Strings.isNullOrEmpty(json)) {
+ return null;
+ }
+
+ try {
+ return objectMapper.readValue(json, clazz);
+ } catch (Exception e) {
+ log.error("Parse object exception, jsonStr: {}, class: {}", json, clazz, e);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
This log entry depends on a [user-provided value](2).
This log entry depends on a [user-provided value](3).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4875)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/FileUtils.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.utils;
+
+import static org.apache.dolphinscheduler.spi.constants.Constants.FORMAT_S_S;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.SystemUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.FileAlreadyExistsException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Set;
+
+import lombok.NonNull;
+import lombok.experimental.UtilityClass;
+import lombok.extern.slf4j.Slf4j;
+
+@UtilityClass
+@Slf4j
+public class FileUtils {
+
+ public static final String KUBE_CONFIG_FILE = "config";
+
+ private static final Set<PosixFilePermission> PERMISSION_755 = PosixFilePermissions.fromString("rwxr-xr-x");
+
+ /**
+ * absolute path of kubernetes configuration file
+ *
+ * @param execPath
+ * @return
+ */
+ public static String getKubeConfigPath(String execPath) {
+ return String.format(FORMAT_S_S, execPath, KUBE_CONFIG_FILE);
+ }
+
+ public static void createFileWith755(@NonNull Path path) throws IOException {
+ if (SystemUtils.IS_OS_WINDOWS) {
+ Files.createFile(path);
+ } else {
+ Files.createFile(path);
+ Files.setPosixFilePermissions(path, PERMISSION_755);
+ }
+ }
+
+ public static void createDirectoryWith755(@NonNull Path path) throws IOException {
+ if (path.toFile().exists()) {
+ return;
+ }
+ if (OSUtils.isWindows()) {
+ Files.createDirectories(path);
+ } else {
+ Path parent = path.getParent();
+ if (parent != null && !parent.toFile().exists()) {
+ createDirectoryWith755(parent);
+ }
+
+ try {
+ Files.createDirectory(path);
+ Files.setPosixFilePermissions(path, PERMISSION_755);
+ } catch (FileAlreadyExistsException fileAlreadyExistsException) {
+ // Catch the FileAlreadyExistsException here to avoid create the same parent directory in parallel
+ log.debug("The directory: {} already exists", path);
Review Comment:
## Log Injection
This log entry depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4874)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/FileUtils.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.utils;
+
+import static org.apache.dolphinscheduler.spi.constants.Constants.FORMAT_S_S;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.SystemUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.FileAlreadyExistsException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Set;
+
+import lombok.NonNull;
+import lombok.experimental.UtilityClass;
+import lombok.extern.slf4j.Slf4j;
+
+@UtilityClass
+@Slf4j
+public class FileUtils {
+
+ public static final String KUBE_CONFIG_FILE = "config";
+
+ private static final Set<PosixFilePermission> PERMISSION_755 = PosixFilePermissions.fromString("rwxr-xr-x");
+
+ /**
+ * absolute path of kubernetes configuration file
+ *
+ * @param execPath
+ * @return
+ */
+ public static String getKubeConfigPath(String execPath) {
+ return String.format(FORMAT_S_S, execPath, KUBE_CONFIG_FILE);
+ }
+
+ public static void createFileWith755(@NonNull Path path) throws IOException {
+ if (SystemUtils.IS_OS_WINDOWS) {
+ Files.createFile(path);
+ } else {
+ Files.createFile(path);
+ Files.setPosixFilePermissions(path, PERMISSION_755);
+ }
+ }
+
+ public static void createDirectoryWith755(@NonNull Path path) throws IOException {
+ if (path.toFile().exists()) {
+ return;
+ }
+ if (OSUtils.isWindows()) {
+ Files.createDirectories(path);
+ } else {
+ Path parent = path.getParent();
+ if (parent != null && !parent.toFile().exists()) {
+ createDirectoryWith755(parent);
+ }
+
+ try {
+ Files.createDirectory(path);
Review Comment:
## Uncontrolled data used in path expression
This path depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4869)
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/utils/JSONUtils.java:
##########
@@ -0,0 +1,427 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.spi.utils;
+
+import static com.fasterxml.jackson.databind.DeserializationFeature.ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT;
+import static com.fasterxml.jackson.databind.DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES;
+import static com.fasterxml.jackson.databind.DeserializationFeature.READ_UNKNOWN_ENUM_VALUES_AS_NULL;
+import static com.fasterxml.jackson.databind.MapperFeature.REQUIRE_SETTERS_FOR_GETTERS;
+import static com.fasterxml.jackson.databind.SerializationFeature.FAIL_ON_EMPTY_BEANS;
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.dolphinscheduler.spi.constants.DateConstants.YYYY_MM_DD_HH_MM_SS;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.time.LocalDateTime;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TimeZone;
+
+import javax.annotation.Nullable;
+
+import lombok.extern.slf4j.Slf4j;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectWriter;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.json.JsonMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fasterxml.jackson.databind.node.TextNode;
+import com.fasterxml.jackson.databind.type.CollectionType;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import com.google.common.base.Strings;
+
+@Slf4j
+public final class JSONUtils {
+
+ static {
+ log.info("init timezone: {}", TimeZone.getDefault());
+ }
+
+ private static final ObjectMapper objectMapper = JsonMapper.builder()
+ .configure(FAIL_ON_UNKNOWN_PROPERTIES, false)
+ .configure(ACCEPT_EMPTY_ARRAY_AS_NULL_OBJECT, true)
+ .configure(READ_UNKNOWN_ENUM_VALUES_AS_NULL, true)
+ .configure(REQUIRE_SETTERS_FOR_GETTERS, true)
+ .configure(FAIL_ON_EMPTY_BEANS, false)
+ .addModule(new JavaTimeModule())
+ .addModule(new SimpleModule()
+ .addSerializer(LocalDateTime.class, new LocalDateTimeSerializer())
+ .addDeserializer(LocalDateTime.class, new LocalDateTimeDeserializer()))
+ .defaultTimeZone(TimeZone.getDefault())
+ .defaultDateFormat(new SimpleDateFormat(YYYY_MM_DD_HH_MM_SS))
+ .build();
+
+ private JSONUtils() {
+ throw new UnsupportedOperationException("Construct JSONUtils");
+ }
+
+ public static synchronized void setTimeZone(TimeZone timeZone) {
+ objectMapper.setTimeZone(timeZone);
+ }
+
+ public static ArrayNode createArrayNode() {
+ return objectMapper.createArrayNode();
+ }
+
+ public static ObjectNode createObjectNode() {
+ return objectMapper.createObjectNode();
+ }
+
+ public static JsonNode toJsonNode(Object obj) {
+ return objectMapper.valueToTree(obj);
+ }
+
+ /**
+ * json representation of object
+ *
+ * @param object object
+ * @param feature feature
+ * @return object to json string
+ */
+ public static String toJsonString(Object object, SerializationFeature feature) {
+ try {
+ ObjectWriter writer = objectMapper.writer(feature);
+ return writer.writeValueAsString(object);
+ } catch (Exception e) {
+ log.error("object to json exception!", e);
+ }
+
+ return null;
+ }
+
+ /**
+ * This method deserializes the specified Json into an object of the specified class. It is not
+ * suitable to use if the specified class is a generic type since it will not have the generic
+ * type information because of the Type Erasure feature of Java. Therefore, this method should not
+ * be used if the desired type is a generic type. Note that this method works fine if the any of
+ * the fields of the specified object are generics, just the object itself should not be a
+ * generic type.
+ *
+ * @param json the string from which the object is to be deserialized
+ * @param clazz the class of T
+ * @param <T> T
+ * @return an object of type T from the string
+ * classOfT
+ */
+ public static @Nullable <T> T parseObject(String json, Class<T> clazz) {
+ if (Strings.isNullOrEmpty(json)) {
+ return null;
+ }
+
+ try {
+ return objectMapper.readValue(json, clazz);
+ } catch (Exception e) {
+ log.error("Parse object exception, jsonStr: {}, class: {}", json, clazz, e);
+ }
+ return null;
+ }
+
+ /**
+ * deserialize
+ *
+ * @param src byte array
+ * @param clazz class
+ * @param <T> deserialize type
+ * @return deserialize type
+ */
+ public static <T> T parseObject(byte[] src, Class<T> clazz) {
+ if (src == null) {
+ return null;
+ }
+ String json = new String(src, UTF_8);
+ return parseObject(json, clazz);
+ }
+
+ /**
+ * json to list
+ *
+ * @param json json string
+ * @param clazz class
+ * @param <T> T
+ * @return list
+ */
+ public static <T> List<T> toList(String json, Class<T> clazz) {
+ if (Strings.isNullOrEmpty(json)) {
+ return Collections.emptyList();
+ }
+
+ try {
+ CollectionType listType = objectMapper.getTypeFactory().constructCollectionType(ArrayList.class, clazz);
+ return objectMapper.readValue(json, listType);
+ } catch (Exception e) {
+ log.error("parse list exception!", e);
+ }
+
+ return Collections.emptyList();
+ }
+
+ /**
+ * check json object valid
+ *
+ * @param json json
+ * @return true if valid
+ */
+ public static boolean checkJsonValid(String json) {
+ return checkJsonValid(json, true);
+ }
+
+ public static boolean checkJsonValid(String json, Boolean logFlag) {
+ if (Strings.isNullOrEmpty(json)) {
+ return false;
+ }
+
+ try {
+ objectMapper.readTree(json);
+ return true;
+ } catch (IOException e) {
+ if (logFlag)
+ log.error("check json object valid exception!", e);
+ }
+
+ return false;
+ }
+
+ /**
+ * Method for finding a JSON Object field with specified name in this
+ * node or its child nodes, and returning value it has.
+ * If no matching field is found in this node or its descendants, returns null.
+ *
+ * @param jsonNode json node
+ * @param fieldName Name of field to look for
+ * @return Value of first matching node found, if any; null if none
+ */
+ public static String findValue(JsonNode jsonNode, String fieldName) {
+ JsonNode node = jsonNode.findValue(fieldName);
+
+ if (node == null) {
+ return null;
+ }
+
+ return node.asText();
+ }
+
+ /**
+ * json to map
+ * {@link #toMap(String, Class, Class)}
+ *
+ * @param json json
+ * @return json to map
+ */
+ public static Map<String, String> toMap(String json) {
+ return parseObject(json, new TypeReference<Map<String, String>>() {
+ });
+ }
+
+ /**
+ * json to map
+ *
+ * @param json json
+ * @param classK classK
+ * @param classV classV
+ * @param <K> K
+ * @param <V> V
+ * @return to map
+ */
+ public static <K, V> Map<K, V> toMap(String json, Class<K> classK, Class<V> classV) {
Review Comment:
## Useless parameter
The parameter 'classV' is never used.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/4871)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS merged PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "EricGao888 (via GitHub)" <gi...@apache.org>.
EricGao888 commented on PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#issuecomment-2367110859
Seems there were some network issues when I clicked `Approve`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#issuecomment-2366262206
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=16613) **Quality Gate failed**
Failed conditions
 [13.5% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=16613&metric=new_coverage&view=list) (required ≥ 60%)
 [11.7% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=16613&metric=new_duplicated_lines_density&view=list) (required ≤ 10%)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=16613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on code in PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#discussion_r1767829581
##########
pom.xml:
##########
@@ -477,64 +507,6 @@
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${exec-maven-plugin.version}</version>
- <executions>
Review Comment:
Move docker build logic to the `dolphinscheduler-dist` module since docker build logic refactored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on code in PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#discussion_r1767820276
##########
dolphinscheduler-dist/src/main/assembly/assembly-plugins.sh:
##########
@@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -xeo pipefail
+
+DIST_DIR="$(pwd)/target"
+BIN_TAR_FILE="$DIST_DIR/apache-dolphinscheduler-*-bin.tar.gz"
+if [ ! -f $BIN_TAR_FILE ]; then
+ echo "$BIN_TAR_FILE not found!!!"
+ exit 1
+fi
+
+cd $DIST_DIR && tar -zxf apache-dolphinscheduler-*-bin.tar.gz
+cd $DIST_DIR/apache-dolphinscheduler-*-bin
+BIN_DIR=$(pwd)
+
+# move *-plugins/target/*-plugin/target/*.jar to *-plugins/
+PLUGINS_PATH=(
+alert-plugins
+datasource-plugins
+storage-plugins
+task-plugins
+)
+
+for plugin_path in ${PLUGINS_PATH[@]}
+do
+ cd $BIN_DIR/plugins/$plugin_path
+ find ./* -name "*.jar" | xargs -I {} mv {} ./
+ ls -d */ | xargs -I {} rm -rf {}
+done
Review Comment:
We should use maven assembly plugin to manage the plugin directory
##########
dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-prometheus/src/main/java/org/apache/dolphinscheduler/plugin/alert/prometheus/PrometheusAlertSender.java:
##########
@@ -68,7 +67,7 @@ public AlertResult sendMessage(AlertData alertData) {
log.error("Send prometheus alert manager alert error", e);
alertResult = new AlertResult();
alertResult.setSuccess(false);
- alertResult.setMessage(ExceptionUtils.getMessage(e));
Review Comment:
Please revert this change, we need to use `ExceptionUtils` to get the exception stack.
##########
dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/bin/start.sh:
##########
@@ -18,11 +18,12 @@
set -eo pipefail
BIN_DIR=$(dirname $(readlink -f "$0"))
-DOLPHINSCHEDULER_HOME=$(cd ${BIN_DIR}/..;pwd)
+DOLPHINSCHEDULER_HOME=$(cd ${BIN_DIR}/../..;pwd)
Review Comment:
Why change this value?
##########
pom.xml:
##########
@@ -477,64 +507,6 @@
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${exec-maven-plugin.version}</version>
- <executions>
Review Comment:
Why remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#issuecomment-2362713605
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=16613) **Quality Gate failed**
Failed conditions
 [13.5% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=16613&metric=new_coverage&view=list) (required ≥ 60%)
 [11.7% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=16613&metric=new_duplicated_lines_density&view=list) (required ≤ 10%)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=16613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on code in PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#discussion_r1767836256
##########
dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-prometheus/src/main/java/org/apache/dolphinscheduler/plugin/alert/prometheus/PrometheusAlertSender.java:
##########
@@ -68,7 +67,7 @@ public AlertResult sendMessage(AlertData alertData) {
log.error("Send prometheus alert manager alert error", e);
alertResult = new AlertResult();
alertResult.setSuccess(false);
- alertResult.setMessage(ExceptionUtils.getMessage(e));
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#issuecomment-2362717119
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=16613) **Quality Gate failed**
Failed conditions
 [13.5% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=16613&metric=new_coverage&view=list) (required ≥ 60%)
 [11.7% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=16613&metric=new_duplicated_lines_density&view=list) (required ≤ 10%)
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=16613) (required ≥ A)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=16613)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on code in PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#discussion_r1767828229
##########
dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-prometheus/src/main/java/org/apache/dolphinscheduler/plugin/alert/prometheus/PrometheusAlertSender.java:
##########
@@ -68,7 +67,7 @@ public AlertResult sendMessage(AlertData alertData) {
log.error("Send prometheus alert manager alert error", e);
alertResult = new AlertResult();
alertResult.setSuccess(false);
- alertResult.setMessage(ExceptionUtils.getMessage(e));
Review Comment:
We've already print the stack trace in L68. I think we don't need to print stack trace here. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-38] Add plugin directory in binary package [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on code in PR #16613:
URL: https://github.com/apache/dolphinscheduler/pull/16613#discussion_r1767831049
##########
dolphinscheduler-dist/src/main/assembly/assembly-plugins.sh:
##########
@@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -xeo pipefail
+
+DIST_DIR="$(pwd)/target"
+BIN_TAR_FILE="$DIST_DIR/apache-dolphinscheduler-*-bin.tar.gz"
+if [ ! -f $BIN_TAR_FILE ]; then
+ echo "$BIN_TAR_FILE not found!!!"
+ exit 1
+fi
+
+cd $DIST_DIR && tar -zxf apache-dolphinscheduler-*-bin.tar.gz
+cd $DIST_DIR/apache-dolphinscheduler-*-bin
+BIN_DIR=$(pwd)
+
+# move *-plugins/target/*-plugin/target/*.jar to *-plugins/
+PLUGINS_PATH=(
+alert-plugins
+datasource-plugins
+storage-plugins
+task-plugins
+)
+
+for plugin_path in ${PLUGINS_PATH[@]}
+do
+ cd $BIN_DIR/plugins/$plugin_path
+ find ./* -name "*.jar" | xargs -I {} mv {} ./
+ ls -d */ | xargs -I {} rm -rf {}
+done
Review Comment:
Assemby packaged directory:
- plugins
- alert-plugins
- dolphinscheduler-alert-dingtalk/taget/dolphinscheduler-alert-dingtalk-shade.jar
Directory processed with shell:
- plugins
- alert-plugins
- dolphinscheduler-alert-dingtalk-shade.jar
Assemby fuzzy matching directory can't remove the file path, so it needs secondary processing with shell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org