You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/07/18 07:06:24 UTC
[GitHub] [incubator-seatunnel] Hisoka-X opened a new pull request, #2193: [Core][Starter] Change jar connector load logic
Hisoka-X opened a new pull request, #2193:
URL: https://github.com/apache/incubator-seatunnel/pull/2193
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
1. Change connector load order, first from classpath, then jar file.
2. Default use engine's current thread classloader to load, avoid different classloader load same class, will cause cast error if not.
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on a diff in pull request #2193: [Core][Starter] Change jar connector load logic
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #2193:
URL: https://github.com/apache/incubator-seatunnel/pull/2193#discussion_r923058604
##########
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:
##########
@@ -40,15 +43,34 @@
import java.util.Optional;
import java.util.ServiceLoader;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.BiConsumer;
import java.util.stream.Collectors;
public abstract class AbstractPluginDiscovery<T> implements PluginDiscovery<T> {
private static final Logger LOGGER = LoggerFactory.getLogger(AbstractPluginDiscovery.class);
private final Path pluginDir;
+ /**
+ * Add jar url to classloader. The different engine should have different logic to add url into
+ * their own classloader
+ */
+ private BiConsumer<ClassLoader, URL> addURLToClassLoader = (classLoader, url) -> {
+ if (classLoader instanceof URLClassLoader) {
+ ReflectionUtils.invoke(classLoader, "addURL", url);
+ } else {
+ throw new UnsupportedOperationException("can't support custom load jar");
+ }
+ };
+
protected final ConcurrentHashMap<PluginIdentifier, Optional<URL>> pluginJarPath =
- new ConcurrentHashMap<>(Common.COLLECTION_SIZE);
+ new ConcurrentHashMap<>(Common.COLLECTION_SIZE);
+
+ public AbstractPluginDiscovery(String pluginSubDir, BiConsumer<ClassLoader, URL> addURLToClassloader) {
Review Comment:
It's better to move the addURL logic to this class, and the other class doesn't need to submit this parameter to tell the AbstractPluginDiscovery how to add URL, since we don't expose the classloader to them, so they don't know which classloader we will use.
```suggestion
public AbstractPluginDiscovery(String pluginSubDir) {
```
##########
seatunnel-core/seatunnel-flink-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/AbstractPluginExecuteProcessor.java:
##########
@@ -41,6 +45,17 @@
protected static final String ENGINE_TYPE = "seatunnel";
protected static final String PLUGIN_NAME = "plugin_name";
+ protected final BiConsumer<ClassLoader, URL> addUrlToClassloader = (classLoader, url) -> {
+ if (classLoader.getClass().getName().endsWith("SafetyNetWrapperClassLoader")) {
+ URLClassLoader c = (URLClassLoader) ReflectionUtils.getField(classLoader, "inner").get();
+ ReflectionUtils.invoke(c, "addURL", url);
Review Comment:
Why not directly use c.addURL
```suggestion
c.addURL(url);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun merged pull request #2193: [Core][Starter] Change jar connector load logic
Posted by GitBox <gi...@apache.org>.
ruanwenjun merged PR #2193:
URL: https://github.com/apache/incubator-seatunnel/pull/2193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on a diff in pull request #2193: [Core][Starter] Change jar connector load logic
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #2193:
URL: https://github.com/apache/incubator-seatunnel/pull/2193#discussion_r924634574
##########
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:
##########
@@ -40,15 +43,34 @@
import java.util.Optional;
import java.util.ServiceLoader;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.BiConsumer;
import java.util.stream.Collectors;
public abstract class AbstractPluginDiscovery<T> implements PluginDiscovery<T> {
private static final Logger LOGGER = LoggerFactory.getLogger(AbstractPluginDiscovery.class);
private final Path pluginDir;
+ /**
+ * Add jar url to classloader. The different engine should have different logic to add url into
+ * their own classloader
+ */
+ private BiConsumer<ClassLoader, URL> addURLToClassLoader = (classLoader, url) -> {
+ if (classLoader instanceof URLClassLoader) {
+ ReflectionUtils.invoke(classLoader, "addURL", url);
+ } else {
+ throw new UnsupportedOperationException("can't support custom load jar");
+ }
+ };
+
protected final ConcurrentHashMap<PluginIdentifier, Optional<URL>> pluginJarPath =
- new ConcurrentHashMap<>(Common.COLLECTION_SIZE);
+ new ConcurrentHashMap<>(Common.COLLECTION_SIZE);
+
+ public AbstractPluginDiscovery(String pluginSubDir, BiConsumer<ClassLoader, URL> addURLToClassloader) {
Review Comment:
Anyway, I approve this first, since we need to fix #2212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on a diff in pull request #2193: [Core][Starter] Change jar connector load logic
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #2193:
URL: https://github.com/apache/incubator-seatunnel/pull/2193#discussion_r924610450
##########
seatunnel-core/seatunnel-flink-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/AbstractPluginExecuteProcessor.java:
##########
@@ -41,6 +45,17 @@
protected static final String ENGINE_TYPE = "seatunnel";
protected static final String PLUGIN_NAME = "plugin_name";
+ protected final BiConsumer<ClassLoader, URL> addUrlToClassloader = (classLoader, url) -> {
+ if (classLoader.getClass().getName().endsWith("SafetyNetWrapperClassLoader")) {
+ URLClassLoader c = (URLClassLoader) ReflectionUtils.getField(classLoader, "inner").get();
+ ReflectionUtils.invoke(c, "addURL", url);
Review Comment:
OK, I understand, we use reflect to break this 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on a diff in pull request #2193: [Core][Starter] Change jar connector load logic
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on code in PR #2193:
URL: https://github.com/apache/incubator-seatunnel/pull/2193#discussion_r923077654
##########
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:
##########
@@ -40,15 +43,34 @@
import java.util.Optional;
import java.util.ServiceLoader;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.BiConsumer;
import java.util.stream.Collectors;
public abstract class AbstractPluginDiscovery<T> implements PluginDiscovery<T> {
private static final Logger LOGGER = LoggerFactory.getLogger(AbstractPluginDiscovery.class);
private final Path pluginDir;
+ /**
+ * Add jar url to classloader. The different engine should have different logic to add url into
+ * their own classloader
+ */
+ private BiConsumer<ClassLoader, URL> addURLToClassLoader = (classLoader, url) -> {
+ if (classLoader instanceof URLClassLoader) {
+ ReflectionUtils.invoke(classLoader, "addURL", url);
+ } else {
+ throw new UnsupportedOperationException("can't support custom load jar");
+ }
+ };
+
protected final ConcurrentHashMap<PluginIdentifier, Optional<URL>> pluginJarPath =
- new ConcurrentHashMap<>(Common.COLLECTION_SIZE);
+ new ConcurrentHashMap<>(Common.COLLECTION_SIZE);
+
+ public AbstractPluginDiscovery(String pluginSubDir, BiConsumer<ClassLoader, URL> addURLToClassloader) {
Review Comment:
This BiConsumer mean engine use their classloader to add url into itself. So caller should tell `AbstractPluginDiscovery ` how to add url into their classloader. The `AbstractPluginDiscovery` is without engine. So it can't decide how to add url into classloader( but also have default implement).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on a diff in pull request #2193: [Core][Starter] Change jar connector load logic
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on code in PR #2193:
URL: https://github.com/apache/incubator-seatunnel/pull/2193#discussion_r923073588
##########
seatunnel-core/seatunnel-flink-starter/src/main/java/org/apache/seatunnel/core/starter/flink/execution/AbstractPluginExecuteProcessor.java:
##########
@@ -41,6 +45,17 @@
protected static final String ENGINE_TYPE = "seatunnel";
protected static final String PLUGIN_NAME = "plugin_name";
+ protected final BiConsumer<ClassLoader, URL> addUrlToClassloader = (classLoader, url) -> {
+ if (classLoader.getClass().getName().endsWith("SafetyNetWrapperClassLoader")) {
+ URLClassLoader c = (URLClassLoader) ReflectionUtils.getField(classLoader, "inner").get();
+ ReflectionUtils.invoke(c, "addURL", url);
Review Comment:
`addURL` is protected scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on a diff in pull request #2193: [Core][Starter] Change jar connector load logic
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #2193:
URL: https://github.com/apache/incubator-seatunnel/pull/2193#discussion_r924627895
##########
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:
##########
@@ -40,15 +43,34 @@
import java.util.Optional;
import java.util.ServiceLoader;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.BiConsumer;
import java.util.stream.Collectors;
public abstract class AbstractPluginDiscovery<T> implements PluginDiscovery<T> {
private static final Logger LOGGER = LoggerFactory.getLogger(AbstractPluginDiscovery.class);
private final Path pluginDir;
+ /**
+ * Add jar url to classloader. The different engine should have different logic to add url into
+ * their own classloader
+ */
+ private BiConsumer<ClassLoader, URL> addURLToClassLoader = (classLoader, url) -> {
+ if (classLoader instanceof URLClassLoader) {
+ ReflectionUtils.invoke(classLoader, "addURL", url);
+ } else {
+ throw new UnsupportedOperationException("can't support custom load jar");
+ }
+ };
+
protected final ConcurrentHashMap<PluginIdentifier, Optional<URL>> pluginJarPath =
- new ConcurrentHashMap<>(Common.COLLECTION_SIZE);
+ new ConcurrentHashMap<>(Common.COLLECTION_SIZE);
+
+ public AbstractPluginDiscovery(String pluginSubDir, BiConsumer<ClassLoader, URL> addURLToClassloader) {
Review Comment:
But the classloader is created by `AbstractPluginDiscovery `, the sub engine discovery doesn't know classloader.
If you expose these in constructor, the caller need to understand what this function means, then he needs to know the whole logic how we load the plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org