You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "liugddx (via GitHub)" <gi...@apache.org> on 2025/01/16 09:29:28 UTC

[I] [Feature][Transform] Add python script tool to Transform [seatunnel]

liugddx opened a new issue, #8542:
URL: https://github.com/apache/seatunnel/issues/8542

   ### Search before asking
   
   - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
   
   
   ### Description
   
   Clean data by writing python scripts.
   
   transform {
     Python {
      for key, value in input_dict.items():
           if key == "name":
               cleaned_value = value.strip() if isinstance(value, str) else value
           elif key == "age":
               try:
                   cleaned_value = int(value)
               except ValueError:
                   cleaned_value = None  # 如果无法转换为整数,设置为 None
           elif key == "email":
               cleaned_value = value.lower() if isinstance(value, str) else value
           else:
               cleaned_value = value
   
           result.append(cleaned_value)
     }
   }
   
   ### Usage Scenario
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2647881788

   @liugddx @hailin0 @Hisoka-X 
   
   Give me some advice and I'm ready to do it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2652732810

   Python transform may also need to output CatalogTable structure changes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "BruceWong96 (via GitHub)" <gi...@apache.org>.
BruceWong96 commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2604441699

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "csh0101 (via GitHub)" <gi...@apache.org>.
csh0101 commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2652622686

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2595031510

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2647847804

   I looked into a solution using py4j, and here's how it works. It supports bidirectional communication between java and python clients.
   To fulfill this requirement:
   1:During the startup of the Seatunnel transform, a resident process is initiated from the Java side. The code is as follows
   ```java
   GatewayServer gatewayServer = new GatewayServer(new ExampleJava());
   gatewayServer.start();
   ```
   ExampleJava provides the following encapsulation of basic SeatunnelRow operations。
   2:Now that we have started a Java server, we need to launch a Python client to connect to the Java server
   ```python
    gateway = JavaGateway(
           callback_server_parameters=CallbackServerParameters()
       )
   
       # 获取 Java 对象 其实就是ExampleJava2
       java_app = gateway.entry_point
   ```
   3: How to execute python code in java?
   Suppose we have a getPython() method on the java side that provides user-configured python code
   ```java
    public String getPythonCode(){
           return "import numpy as np;\n" +
                   "print(np.array([1, 2, 3]));print(\"[Python] 这是 Java 传入的代码!java_app.callPythonMethod(1, 1.1)\");java_app.callPythonMethod(1, 1.1)";
       }
   ```
   ```python
    # 在python 处申明java对象
           java_list = gateway.jvm.java.util.ArrayList()
           java_list.append(1)
           java_list.append(2)
           java_app.addList(1,java_list)
           # 在python 处申明java对象
           my_object = gateway.jvm.com.example.MyObject("test", 123)
           java_app.addObject(1,my_object);
           try:
               ## 执行 java段给定的自定义代码
               exec(java_app.getPythonCode())  # 动态执行 Python 代码
           except Exception as e:
               print(f"[Python] 代码执行出错: {e}")
   ```python
   4: The flow chart is as follows
   
   ![Image](https://github.com/user-attachments/assets/3ba8f097-9be0-474f-b31b-61bd9cf15d34)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2647863894

   demo 
   
   <!-- Failed to upload "录屏2025-02-01 21.26.32.mov" -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8542: [Feature][Transform] Add python script tool to Transform
URL: https://github.com/apache/seatunnel/issues/8542


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2604555463

   Please assign it to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2652772115

   Are there any prerequisites for using py4j?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2684959251

   在搞了 在搞了


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-3066302802

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-3092661304

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2658595017

   > Are there any prerequisites for using py4j?
   
   java端 seatunnel会自己引入
   python端 需要服务器提供python环境


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]

Posted by "SimonChou12138 (via GitHub)" <gi...@apache.org>.
SimonChou12138 commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2653275077

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org