You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "liugddx (via GitHub)" <gi...@apache.org> on 2025/01/16 09:29:28 UTC
[I] [Feature][Transform] Add python script tool to Transform [seatunnel]
liugddx opened a new issue, #8542:
URL: https://github.com/apache/seatunnel/issues/8542
### Search before asking
- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
Clean data by writing python scripts.
transform {
Python {
for key, value in input_dict.items():
if key == "name":
cleaned_value = value.strip() if isinstance(value, str) else value
elif key == "age":
try:
cleaned_value = int(value)
except ValueError:
cleaned_value = None # 如果无法转换为整数,设置为 None
elif key == "email":
cleaned_value = value.lower() if isinstance(value, str) else value
else:
cleaned_value = value
result.append(cleaned_value)
}
}
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2647881788
@liugddx @hailin0 @Hisoka-X
Give me some advice and I'm ready to do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2652732810
Python transform may also need to output CatalogTable structure changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "BruceWong96 (via GitHub)" <gi...@apache.org>.
BruceWong96 commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2604441699
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "csh0101 (via GitHub)" <gi...@apache.org>.
csh0101 commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2652622686
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2595031510
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2647847804
I looked into a solution using py4j, and here's how it works. It supports bidirectional communication between java and python clients.
To fulfill this requirement:
1:During the startup of the Seatunnel transform, a resident process is initiated from the Java side. The code is as follows
```java
GatewayServer gatewayServer = new GatewayServer(new ExampleJava());
gatewayServer.start();
```
ExampleJava provides the following encapsulation of basic SeatunnelRow operations。
2:Now that we have started a Java server, we need to launch a Python client to connect to the Java server
```python
gateway = JavaGateway(
callback_server_parameters=CallbackServerParameters()
)
# 获取 Java 对象 其实就是ExampleJava2
java_app = gateway.entry_point
```
3: How to execute python code in java?
Suppose we have a getPython() method on the java side that provides user-configured python code
```java
public String getPythonCode(){
return "import numpy as np;\n" +
"print(np.array([1, 2, 3]));print(\"[Python] 这是 Java 传入的代码!java_app.callPythonMethod(1, 1.1)\");java_app.callPythonMethod(1, 1.1)";
}
```
```python
# 在python 处申明java对象
java_list = gateway.jvm.java.util.ArrayList()
java_list.append(1)
java_list.append(2)
java_app.addList(1,java_list)
# 在python 处申明java对象
my_object = gateway.jvm.com.example.MyObject("test", 123)
java_app.addObject(1,my_object);
try:
## 执行 java段给定的自定义代码
exec(java_app.getPythonCode()) # 动态执行 Python 代码
except Exception as e:
print(f"[Python] 代码执行出错: {e}")
```python
4: The flow chart is as follows

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2647863894
demo
<!-- Failed to upload "录屏2025-02-01 21.26.32.mov" -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8542: [Feature][Transform] Add python script tool to Transform
URL: https://github.com/apache/seatunnel/issues/8542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2604555463
Please assign it to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2652772115
Are there any prerequisites for using py4j?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2684959251
在搞了 在搞了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-3066302802
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-3092661304
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "FuYouJ (via GitHub)" <gi...@apache.org>.
FuYouJ commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2658595017
> Are there any prerequisites for using py4j?
java端 seatunnel会自己引入
python端 需要服务器提供python环境
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Transform] Add python script tool to Transform [seatunnel]
Posted by "SimonChou12138 (via GitHub)" <gi...@apache.org>.
SimonChou12138 commented on issue #8542:
URL: https://github.com/apache/seatunnel/issues/8542#issuecomment-2653275077
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org