You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "corgy-w (via GitHub)" <gi...@apache.org> on 2024/07/26 10:43:53 UTC

[PR] [bug][plugin-discovery] fix multi plugin discovery [seatunnel]

corgy-w opened a new pull request, #7278:
URL: https://github.com/apache/seatunnel/pull/7278

   ### Purpose of this pull request
   issue : #7271 
   
   ### Does this PR introduce _any_ user-facing change?
   
   
   ### How was this patch tested?
   
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
     2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Core] Improve base on plugin name of lookup strategy [seatunnel]

Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN merged PR #7278:
URL: https://github.com/apache/seatunnel/pull/7278


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Improve][Core] Improve base on plugin name of lookup strategy [seatunnel]

Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on PR #7278:
URL: https://github.com/apache/seatunnel/pull/7278#issuecomment-2255439998

   good job @corgy-w 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [bug][plugin-discovery] fix multi plugin discovery [seatunnel]

Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w commented on code in PR #7278:
URL: https://github.com/apache/seatunnel/pull/7278#discussion_r1694645361


##########
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:
##########
@@ -427,20 +429,12 @@ public boolean accept(File pathname) {
                                                         pathname.getName(), pluginJarPrefix);
                                     }
                                 });
-        if (ArrayUtils.isEmpty(targetPluginFiles)) {
+        if (ArrayUtils.isEmpty(targetPluginFiles) || targetPluginFiles == null) {

Review Comment:
   yes, misled by idea editor



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [bug][plugin-discovery] fix multi plugin discovery [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #7278:
URL: https://github.com/apache/seatunnel/pull/7278#discussion_r1694629810


##########
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:
##########
@@ -427,20 +429,12 @@ public boolean accept(File pathname) {
                                                         pathname.getName(), pluginJarPrefix);
                                     }
                                 });
-        if (ArrayUtils.isEmpty(targetPluginFiles)) {
+        if (ArrayUtils.isEmpty(targetPluginFiles) || targetPluginFiles == null) {
             return Optional.empty();
         }
-        if (targetPluginFiles.length > 1) {
-            throw new IllegalArgumentException(
-                    "Found multiple plugin jar: "
-                            + Arrays.stream(targetPluginFiles)
-                                    .map(File::getPath)
-                                    .collect(Collectors.joining(","))
-                            + " for pluginIdentifier: "
-                            + pluginIdentifier);
-        }
         try {
-            URL pluginJarPath = targetPluginFiles[0].toURI().toURL();
+            URL pluginJarPath =
+                    findMostSimlarPluginJarFile(targetPluginFiles, pluginJarPrefix).toURI().toURL();

Review Comment:
   ```suggestion
           if (targetPluginFiles.length == 1) {
               URL pluginJarPath = targetPluginFiles[0].toURI().toURL();
           } else {
               URL pluginJarPath =
                       findMostSimlarPluginJarFile(targetPluginFiles, pluginJarPrefix).toURI().toURL();
           }
   ```



##########
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:
##########
@@ -451,4 +445,105 @@ public boolean accept(File pathname) {
             return Optional.empty();
         }
     }
+
+    private static File findMostSimlarPluginJarFile(
+            File[] targetPluginFiles, String pluginJarPrefix) {
+        String splitRegex = "\\-|\\_|\\.";
+        double maxSimlarity = -Integer.MAX_VALUE;
+        int mostSimlarPluginJarFileIndex = -1;
+        for (int i = 0; i < targetPluginFiles.length; i++) {
+            File file = targetPluginFiles[i];
+            String fileName = file.getName();
+            double similarity =
+                    CosineSimilarityUtil.cosineSimilarity(pluginJarPrefix, fileName, splitRegex);
+            if (similarity > maxSimlarity) {
+                maxSimlarity = similarity;
+                mostSimlarPluginJarFileIndex = i;
+            }
+        }
+        return targetPluginFiles[mostSimlarPluginJarFileIndex];
+    }
+
+    static class CosineSimilarityUtil {
+        public static double cosineSimilarity(String textA, String textB, String splitRegrex) {
+            Set<String> words1 =
+                    new HashSet<>(Arrays.asList(textA.toLowerCase().split(splitRegrex)));
+            Set<String> words2 =
+                    new HashSet<>(Arrays.asList(textB.toLowerCase().split(splitRegrex)));
+            int[] termFrequency1 = calculateTermFrequencyVector(textA, words1, splitRegrex);
+            int[] termFrequency2 = calculateTermFrequencyVector(textB, words2, splitRegrex);
+            return calculateCosineSimilarity(termFrequency1, termFrequency2);
+        }
+
+        public static int[] calculateTermFrequencyVector(

Review Comment:
   ```suggestion
           private static int[] calculateTermFrequencyVector(
   ```



##########
seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java:
##########
@@ -427,20 +429,12 @@ public boolean accept(File pathname) {
                                                         pathname.getName(), pluginJarPrefix);
                                     }
                                 });
-        if (ArrayUtils.isEmpty(targetPluginFiles)) {
+        if (ArrayUtils.isEmpty(targetPluginFiles) || targetPluginFiles == null) {

Review Comment:
   ```suggestion
           if (ArrayUtils.isEmpty(targetPluginFiles)) {
   ```
   `ArrayUtils.isEmpty` already contains null check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org