You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/09/18 08:00:33 UTC
[GitHub] [dolphinscheduler] zhongjiajie opened a new pull request #6269: Init DS python SDK pydolphinscheduler
zhongjiajie opened a new pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269
## Purpose of the pull request
First commit and init for pydolphinscheduler
## Brief change log
Initialization pydolphinscheduler, a python sdk for DolphinScheduler which could define workflow by code
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930948172
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930948172
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-952748529
> > You need to check whether the jar is license-compatible with APL2.0
>
>
>
> @kezhenxu94 How could I check whether it compatible to APL2.0 or not?
Read this https://www.apache.org/legal/resolved.html
According to your comment, py4j is BSD 2.0 so it's compatible with APL2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953671720
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953683143
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-925876738
# How to reproduce
* **Start java gateway**: run `dolphinscheduler-python/src/main/java/server/PythonGatewayServer.java` in your IDEA
* **Run python tutorial**: run python SDK tutorial in https://github.com/apache/dolphinscheduler/pull/6269/files#diff-5561fec6b57cc611bee2b0d8f030965d76bdd202801d9f8a1e2e74c21769bc41
After this two steps, you could see above error and found server are null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#discussion_r737985777
##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/java_gateway.py
##########
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Optional
+
+from py4j.java_collections import JavaMap
+from py4j.java_gateway import JavaGateway, GatewayParameters
+
+from pydolphinscheduler.constants import JavaGatewayDefault
+
+
+def launch_gateway() -> JavaGateway:
+ # TODO Note that automatic conversion makes calling Java methods slightly less efficient because
+ # in the worst case, Py4J needs to go through all registered converters for all parameters.
+ # This is why automatic conversion is disabled by default.
+ gateway = JavaGateway(gateway_parameters=GatewayParameters(auto_convert=True))
+ return gateway
+
+
+def gateway_result_checker(
+ result: JavaMap,
+ msg_check: Optional[str] = JavaGatewayDefault.RESULT_MESSAGE_SUCCESS
+) -> Any:
+ if result[JavaGatewayDefault.RESULT_STATUS_KEYWORD].toString() != \
+ JavaGatewayDefault.RESULT_STATUS_SUCCESS:
+ raise RuntimeError(f"Failed when try to got result for java gateway")
+ if msg_check is not None and result[JavaGatewayDefault.RESULT_MESSAGE_KEYWORD] != msg_check:
+ raise ValueError(f"Get result state not success.")
+ return result
+
+# try to add decorators bu failed
+# def gateway_result_checker(func, *args, **kwargs):
Review comment:
Good catch π ! I forgot about this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie edited a comment on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie edited a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953469430
When I run python define code, I could see workflow was created and process instance start but failed. The screenshot as below
And I see some error log look like
```log
2021-10-28 11:17:07.160 INFO 30083 --- [tp905404580-123] o.a.d.a.service.impl.LoggerServiceImpl : log host : 192.168.1.102 , logPath : , logServer port : 50051
2021-10-28 11:17:07.160 INFO 30083 --- [tp905404580-123] o.a.d.service.log.LogClientService : roll view log, host : 192.168.1.102, port : 50051, path , skipLineNum 0 ,limit 1000
2021-10-28 11:17:07.193 WARN 30083 --- [tp905404580-123] o.a.d.remote.NettyRemotingClient : connect to Host{address='192.168.1.102:50051', ip='192.168.1.102', port=50051} error
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-926599723
The PythonGatewayServer is run by spring and it can manage beans by annotations like @Autowired,
so if you new the PythonGatewayServer by yourself, which is out of Spring control, you will get some null beans.
you can have a try like `GatewayServer server = new GatewayServer(this);` instead of new PythonGatewayServer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-931135514
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-952776021
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-926540418
FYI, after I add commit https://github.com/apache/dolphinscheduler/pull/6269/commits/a4d84aa3091fdddd247be6de404cd4a00028f648
we could install python package `pydolphinscheduler` locally by run command `python setup.py install` in dir `dolphinscheduler/dolphinscheduler-python/pydolphinscheduler`
After command finish, we could run `tutorial.py` directly by
```shell
cd dolphinscheduler-python/pydolphinscheduler
python example/tutorial.py
```
to make a quick look how `pydolphinscheduler` work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-926541644
cc: @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953469055
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lenboo merged pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
lenboo merged pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930132739
After commit https://github.com/apache/dolphinscheduler/pull/6269/commits/37bd41afea64e45e2c37afb36a42fd8f505f56be add, we fix error while adding new task and change task attribute(including task name).
But I found out a bug when update workflow tasks, I create an new issue to track it https://github.com/apache/dolphinscheduler/issues/6418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930952492
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6269](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (18b3157) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/a9d805d79117047e571deeb52fbf96531c19674b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a9d805d) will **decrease** coverage by `0.06%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## dev #6269 +/- ##
============================================
- Coverage 38.82% 38.76% -0.07%
- Complexity 3219 3220 +1
============================================
Files 645 645
Lines 25710 25736 +26
Branches 2785 2790 +5
============================================
- Hits 9982 9976 -6
- Misses 14806 14842 +36
+ Partials 922 918 -4
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Ξ | |
|---|---|---|
| [...scheduler/api/service/impl/ProjectServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9Qcm9qZWN0U2VydmljZUltcGwuamF2YQ==) | `74.03% <0.00%> (-3.88%)` | :arrow_down: |
| [...inscheduler/api/service/impl/QueueServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9RdWV1ZVNlcnZpY2VJbXBsLmphdmE=) | `64.81% <0.00%> (-7.36%)` | :arrow_down: |
| [...nscheduler/api/service/impl/TenantServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9UZW5hbnRTZXJ2aWNlSW1wbC5qYXZh) | `70.27% <0.00%> (-4.02%)` | :arrow_down: |
| [.../org/apache/dolphinscheduler/common/Constants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL0NvbnN0YW50cy5qYXZh) | `75.00% <ΓΈ> (ΓΈ)` | |
| [...r/plugin/registry/zookeeper/ZookeeperRegistry.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS1wbHVnaW4vZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS16b29rZWVwZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3JlZ2lzdHJ5L3pvb2tlZXBlci9ab29rZWVwZXJSZWdpc3RyeS5qYXZh) | `39.00% <0.00%> (-5.68%)` | :arrow_down: |
| [...inscheduler/common/thread/ThreadPoolExecutors.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3RocmVhZC9UaHJlYWRQb29sRXhlY3V0b3JzLmphdmE=) | `21.15% <0.00%> (+1.92%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Ξ = absolute <relative> (impact)`, `ΓΈ = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [a9d805d...18b3157](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-931006806
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-931006806
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-952740660
> You need to check whether the jar is license-compatible with APL2.0
@kezhenxu94 How could I check whether it compatible to APL2.0 or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953469430
When I run python define code, I could see workflow was created and process instance start but failed. The screenshot as below
And I see some error log look like
```log
2021-10-28 11:17:07.160 INFO 30083 --- [tp905404580-123] o.a.d.a.service.impl.LoggerServiceImpl : log host : 192.168.1.102 , logPath : , logServer port : 50051
2021-10-28 11:17:07.160 INFO 30083 --- [tp905404580-123] o.a.d.service.log.LogClientService : roll view log, host : 192.168.1.102, port : 50051, path , skipLineNum 0 ,limit 1000
2021-10-28 11:17:07.193 WARN 30083 --- [tp905404580-123] o.a.d.remote.NettyRemotingClient : connect to Host{address='192.168.1.102:50051', ip='192.168.1.102', port=50051} error
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.1.102:50051
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_231]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_231]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[netty-all-4.1.53.Final.jar:4.1.53.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-all-4.1.53.Final.jar:4.1.53.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[netty-all-4.1.53.Final.jar:4.1.53.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-all-4.1.53.Final.jar:4.1.53.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-all-4.1.53.Final.jar:4.1.53.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-all-4.1.53.Final.jar:4.1.53.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-all-4.1.53.Final.jar:4.1.53.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-all-4.1.53.Final.jar:4.1.53.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_231]
2021-10-28 11:17:07.206 ERROR 30083 --- [tp905404580-123] o.a.d.service.log.LogClientService : roll view log error
org.apache.dolphinscheduler.remote.exceptions.RemotingException: connect to : Host{address='192.168.1.102:50051', ip='192.168.1.102', port=50051} fail
at org.apache.dolphinscheduler.remote.NettyRemotingClient.sendSync(NettyRemotingClient.java:257) ~[classes/:na]
at org.apache.dolphinscheduler.service.log.LogClientService.rollViewLog(LogClientService.java:94) ~[classes/:na]
at org.apache.dolphinscheduler.api.service.impl.LoggerServiceImpl.queryLog(LoggerServiceImpl.java:108) [classes/:na]
at org.apache.dolphinscheduler.api.controller.LoggerController.queryLog(LoggerController.java:82) [classes/:na]
at org.apache.dolphinscheduler.api.controller.LoggerController$$FastClassBySpringCGLIB$$95a745ef.invoke(<generated>) [classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:752) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.apache.dolphinscheduler.api.aspect.AccessLogAspect.doAround(AccessLogAspect.java:87) [classes/:na]
at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_231]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_231]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) [spring-aop-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.apache.dolphinscheduler.api.controller.LoggerController$$EnhancerBySpringCGLIB$$2a60e4db.queryLog(<generated>) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_231]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_231]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_231]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_231]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1633) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at com.github.xiaoymin.swaggerbootstrapui.filter.SecurityBasicAuthFilter.doFilter(SecurityBasicAuthFilter.java:84) [swagger-bootstrap-ui-1.9.3.jar:na]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at com.github.xiaoymin.swaggerbootstrapui.filter.ProductionSecurityFilter.doFilter(ProductionSecurityFilter.java:53) [swagger-bootstrap-ui-1.9.3.jar:na]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:97) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:94) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.1.19.RELEASE.jar:5.1.19.RELEASE]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) [jetty-security-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) [jetty-servlet-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:766) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.Server.handle(Server.java:516) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) [jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) ~[jetty-server-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) ~[jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773) ~[jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905) ~[jetty-util-9.4.33.v20201020.jar:9.4.33.v20201020]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_231]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-952790214
It's good to go now, PTAL @dailidong @CalvinKirs @lenboo @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#discussion_r714713021
##########
File path: dolphinscheduler-python/src/main/java/server/PythonGatewayServer.java
##########
@@ -36,16 +37,40 @@
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.FilterType;
+import org.springframework.transaction.annotation.EnableTransactionManagement;
import py4j.GatewayServer;
+import javax.annotation.PostConstruct;
import java.util.Map;
-@SpringBootApplication
+@ComponentScan(value = "org.apache.dolphinscheduler", excludeFilters = {
+ @ComponentScan.Filter(type = FilterType.REGEX, pattern = {
+ "org.apache.dolphinscheduler.server.master.*",
+ "org.apache.dolphinscheduler.server.monitor.*",
+ "org.apache.dolphinscheduler.server.log.*"
+ })
+})
+@EnableTransactionManagement
public class PythonGatewayServer {
// TODO could this work in py4j
@Autowired
private ProcessDefinitionMapper processDefinitionMapper;
+ @Autowired
+ private ProjectService projectService;
+
+ @Autowired
+ private TenantService tenantService;
+
+ @Autowired
+ private ExecutorService executorService;
+
+ @Autowired
+ private ProcessDefinitionService processDefinitionService;
+
+ @Autowired
+ private TaskDefinitionService taskDefinitionService;
+
Review comment:
It's odd that ALL server and mapper are `null` when I call from python client, and I could not get the right result from this server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930200515
> Could I code style check and auto fix in my local env?
I found out we have check-style file and idea plugin in https://dolphinscheduler.apache.org/zh-cn/community/development/pull-request.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930185763
Could I code style check and auto fix in my local env?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#discussion_r714713021
##########
File path: dolphinscheduler-python/src/main/java/server/PythonGatewayServer.java
##########
@@ -36,16 +37,40 @@
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.FilterType;
+import org.springframework.transaction.annotation.EnableTransactionManagement;
import py4j.GatewayServer;
+import javax.annotation.PostConstruct;
import java.util.Map;
-@SpringBootApplication
+@ComponentScan(value = "org.apache.dolphinscheduler", excludeFilters = {
+ @ComponentScan.Filter(type = FilterType.REGEX, pattern = {
+ "org.apache.dolphinscheduler.server.master.*",
+ "org.apache.dolphinscheduler.server.monitor.*",
+ "org.apache.dolphinscheduler.server.log.*"
+ })
+})
+@EnableTransactionManagement
public class PythonGatewayServer {
// TODO could this work in py4j
@Autowired
private ProcessDefinitionMapper processDefinitionMapper;
+ @Autowired
+ private ProjectService projectService;
+
+ @Autowired
+ private TenantService tenantService;
+
+ @Autowired
+ private ExecutorService executorService;
+
+ @Autowired
+ private ProcessDefinitionService processDefinitionService;
+
+ @Autowired
+ private TaskDefinitionService taskDefinitionService;
+
Review comment:
It's odd that ALL server and mapper are `null` when I call from python client, and I could not get the right result from this server. My IDEA show as below when I debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-952757502
Thanks, BTW your response as fast as usual
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#discussion_r737986598
##########
File path: dolphinscheduler-python/pom.xml
##########
@@ -0,0 +1,61 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+<modelVersion>4.0.0</modelVersion>
+<parent>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+</parent>
+<artifactId>dolphinscheduler-python</artifactId>
+<name>${project.artifactId}</name>
+<packaging>jar</packaging>
+
+<dependencies>
+ <!-- dolphinscheduler -->
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-api</artifactId>
+ </dependency>
+
+ <!--springboot-->
+ <dependency>
+ <groupId>org.springframework.boot</groupId>
+ <artifactId>spring-boot-starter-web</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.springframework.boot</groupId>
+ <artifactId>spring-boot-starter-tomcat</artifactId>
+ </exclusion>
+ <exclusion>
+ <artifactId>log4j-to-slf4j</artifactId>
+ <groupId>org.apache.logging.log4j</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>net.sf.py4j</groupId>
+ <artifactId>py4j</artifactId>
+ <version>0.10.9</version>
Review comment:
I add this to parent `pom.xml`, but it seem we should still add dependency
```xml
<dependency>
<groupId>net.sf.py4j</groupId>
<artifactId>py4j</artifactId>
</dependency>
```
to python/pom.xml. otherwise it could not find py4j dependent. Am I wrong? @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-926992752
> Some suggestion for you @caishunfeng
Thanks, that's a good suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930982071
Questions: could I add `py4j.jar` to file `known-dependencies.txt` directly? for passing `check-LICENSE` CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-952776021
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#discussion_r715655903
##########
File path: dolphinscheduler-python/src/main/java/org/apache/dolphinscheduler/server/PythonGatewayServer.java
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.server;
+
+import org.apache.dolphinscheduler.api.enums.Status;
+import org.apache.dolphinscheduler.api.service.*;
+import org.apache.dolphinscheduler.common.Constants;
+import org.apache.dolphinscheduler.common.enums.*;
+import org.apache.dolphinscheduler.dao.entity.ProcessDefinition;
+import org.apache.dolphinscheduler.dao.entity.Project;
+import org.apache.dolphinscheduler.dao.entity.Queue;
+import org.apache.dolphinscheduler.dao.entity.User;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.WebApplicationType;
+import org.springframework.boot.builder.SpringApplicationBuilder;
+import org.springframework.context.annotation.ComponentScan;
+import py4j.GatewayServer;
+
+import javax.annotation.PostConstruct;
+import java.util.Map;
+
+@ComponentScan(value = "org.apache.dolphinscheduler")
+public class PythonGatewayServer {
+ // TODO could this work in py4j
+// @Autowired
+// private ProcessDefinitionMapper processDefinitionMapper;
+
+ @Autowired
+ private ProjectService projectService;
+
+ @Autowired
+ private TenantService tenantService;
+
+ @Autowired
+ private ExecutorService executorService;
+
+ @Autowired
+ private ProcessDefinitionService processDefinitionService;
+
+ @Autowired
+ private TaskDefinitionService taskDefinitionService;
+
+
+ private final User userPythonGateway = new User() {
+ {
+ setId(Integer.MAX_VALUE);
+ setUserName("userPythonGateway");
+ setUserType(UserType.ADMIN_USER);
+ }
+ };
+
+ private final Queue queuePythonGateway = new Queue() {
+ {
+ setId(Integer.MAX_VALUE);
+ setQueueName("queuePythonGateway");
+ }
+ };
+
+ public String ping() {
+ return "PONG";
+ }
+
+ // TODO Should we import package in python client side? utils package can but service can not, why
+ // Core api
+ public Map<String, Object> genTaskCodeList(Integer genNum) {
+ return taskDefinitionService.genTaskCodeList(userPythonGateway, genNum);
+ }
+
+ /**
+ * create or update process definition.
+ * If process definition do not exists in Project=`projectCode` would create a new one
+ * If process definition already exists in Project=`projectCode` would update it
+ * All requests
+ * <p>
+ * // * @param loginUser login user
+ *
+ * @param projectCode project code
+ * @param name process definition name
+ * @param description description
+ * @param globalParams global params
+ * @param locations locations for nodes
+ * @param timeout timeout
+ * @param tenantCode tenantCode
+ * @param taskRelationJson relation json for nodes
+ * @param taskDefinitionJson taskDefinitionJson
+ * @return create result code
+ */
+ public Long createOrUpdateProcessDefinition(String projectName,
+ String name,
+ String description,
+ String globalParams,
+ String locations,
+ int timeout,
+ String tenantCode,
+ String taskRelationJson,
+ String taskDefinitionJson) {
+ Project project = (Project) projectService.queryByName(userPythonGateway, name).get(Constants.DATA_LIST);
+ long projectCode = project.getCode();
+ Map<String, Object> verifyProcessDefinitionName = processDefinitionService.verifyProcessDefinitionName(userPythonGateway, projectCode, name);
+
+ if (verifyProcessDefinitionName.get(Constants.STATUS) == Status.SUCCESS) {
+ // update process definition
+// ProcessDefinition processDefinition = processDefinitionMapper.queryByDefineName(projectCode, name);
+// long code = processDefinition.getCode();
+// Map<String, Object> result = processDefinitionService.updateProcessDefinition(userPythonGateway, projectCode, name, code, description, globalParams,
+// locations, timeout, tenantCode, taskRelationJson, taskDefinitionJson);
+// return code;
+ return 1L;
+ } else {
+ // create process definition
+ Map<String, Object> result = processDefinitionService.createProcessDefinition(userPythonGateway, projectCode, name, description, globalParams,
+ locations, timeout, tenantCode, taskRelationJson, taskDefinitionJson);
+ ProcessDefinition processDefinition = (ProcessDefinition) result.get(Constants.DATA_LIST);
+ return processDefinition.getCode();
+ }
+ }
+
+ public void execProcessInstance(long projectCode,
+ long processDefinitionCode,
+ String cronTime,
+ CommandType commandType,
+ FailureStrategy failureStrategy,
+ String startNodeList,
+ TaskDependType taskDependType,
+ WarningType warningType,
+ int warningGroupId,
+ RunMode runMode,
+ Priority processInstancePriority,
+ String workerGroup,
+ Long environmentCode,
+ Integer timeout,
+ Map<String, String> startParams,
+ Integer expectedParallelismNumber) {
+ executorService.execProcessInstance(userPythonGateway,
+ projectCode,
+ processDefinitionCode,
+ cronTime,
+ commandType,
+ failureStrategy,
+ startNodeList,
+ taskDependType,
+ warningType,
+ warningGroupId,
+ runMode,
+ processInstancePriority,
+ workerGroup,
+ environmentCode,
+ timeout, startParams,
+ expectedParallelismNumber);
+ }
+
+
+ // side object
+ public Map<String, Object> createProject(String name, String desc) {
+ return projectService.createProject(userPythonGateway, name, desc);
+ }
+
+ public Map<String, Object> createTenant(String tenantCode, String desc) throws Exception {
+ return tenantService.createTenant(userPythonGateway, tenantCode, queuePythonGateway.getId(), desc);
+ }
+
+ @PostConstruct
+ public void run() {
+ PythonGatewayServer app = new PythonGatewayServer();
+ // app is now the gateway.entry_point
+ // GatewayServer server = new GatewayServer(app,25334); to specific port
+ GatewayServer server = new GatewayServer(app);
Review comment:
```suggestion
GatewayServer server = new GatewayServer(this);
```
Hi @caishunfeng, next time when you try to add suggestions in PR, maybe you could directly [reviewing changes][1] like this comment. I make code reviewed more directly. what more, when PR author accept your review, you would become [co-author][2] after PR merge
[1]: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/reviewing-proposed-changes-in-a-pull-request
[2]: https://docs.github.com/en/github/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter edited a comment on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930952492
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6269](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (67f956a) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/d91801e6f60670915a97d031b21fe16ce98e3e4e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d91801e) will **decrease** coverage by `0.08%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## dev #6269 +/- ##
============================================
- Coverage 38.55% 38.46% -0.09%
+ Complexity 3224 3221 -3
============================================
Files 646 646
Lines 25840 25866 +26
Branches 2799 2804 +5
============================================
- Hits 9963 9950 -13
- Misses 14961 15002 +41
+ Partials 916 914 -2
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Ξ | |
|---|---|---|
| [...scheduler/api/service/impl/ProjectServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9Qcm9qZWN0U2VydmljZUltcGwuamF2YQ==) | `74.03% <0.00%> (-3.88%)` | :arrow_down: |
| [...inscheduler/api/service/impl/QueueServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9RdWV1ZVNlcnZpY2VJbXBsLmphdmE=) | `64.81% <0.00%> (-7.36%)` | :arrow_down: |
| [...nscheduler/api/service/impl/TenantServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9UZW5hbnRTZXJ2aWNlSW1wbC5qYXZh) | `70.27% <0.00%> (-4.02%)` | :arrow_down: |
| [.../org/apache/dolphinscheduler/common/Constants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL0NvbnN0YW50cy5qYXZh) | `75.00% <ΓΈ> (ΓΈ)` | |
| [...ache/dolphinscheduler/server/log/LoggerServer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL2xvZy9Mb2dnZXJTZXJ2ZXIuamF2YQ==) | `78.26% <0.00%> (-8.70%)` | :arrow_down: |
| [...r/plugin/registry/zookeeper/ZookeeperRegistry.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS1wbHVnaW4vZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS16b29rZWVwZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3JlZ2lzdHJ5L3pvb2tlZXBlci9ab29rZWVwZXJSZWdpc3RyeS5qYXZh) | `39.00% <0.00%> (-5.68%)` | :arrow_down: |
| [...er/master/processor/queue/TaskResponseService.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9wcm9jZXNzb3IvcXVldWUvVGFza1Jlc3BvbnNlU2VydmljZS5qYXZh) | `57.31% <0.00%> (-3.66%)` | :arrow_down: |
| [...dolphinscheduler/remote/future/ResponseFuture.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL2Z1dHVyZS9SZXNwb25zZUZ1dHVyZS5qYXZh) | `81.35% <0.00%> (-1.70%)` | :arrow_down: |
| [...er/master/dispatch/host/assign/RandomSelector.java](https://codecov.io/gh/apache/dolphinscheduler/pull/6269/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9ob3N0L2Fzc2lnbi9SYW5kb21TZWxlY3Rvci5qYXZh) | `83.33% <0.00%> (+5.55%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Ξ = absolute <relative> (impact)`, `ΓΈ = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d91801e...67f956a](https://codecov.io/gh/apache/dolphinscheduler/pull/6269?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930132739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie edited a comment on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie edited a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930200515
> Could I code style check and auto fix in my local env?
I found out we have check-style file and idea plugin in https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-931139989
> Questions: could I add `py4j.jar` to file `known-dependencies.txt` directly? for passing `check-LICENSE` CI? py4j license in https://github.com/bartdag/py4j/blob/master/LICENSE.txt
You need to check whether the jar is license-compatible with APL2.0, and add the corresponding license info into the `dolphinscheduler-dist/release-docs/licenses`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-931135514
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie edited a comment on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie edited a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-952790214
It's good to go now, I add example to desc how it work, and add some doc about pydolphinscheduler dev env build and core concept in readme
PTAL @dailidong @CalvinKirs @lenboo @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a change in pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on a change in pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#discussion_r737985777
##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/java_gateway.py
##########
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Optional
+
+from py4j.java_collections import JavaMap
+from py4j.java_gateway import JavaGateway, GatewayParameters
+
+from pydolphinscheduler.constants import JavaGatewayDefault
+
+
+def launch_gateway() -> JavaGateway:
+ # TODO Note that automatic conversion makes calling Java methods slightly less efficient because
+ # in the worst case, Py4J needs to go through all registered converters for all parameters.
+ # This is why automatic conversion is disabled by default.
+ gateway = JavaGateway(gateway_parameters=GatewayParameters(auto_convert=True))
+ return gateway
+
+
+def gateway_result_checker(
+ result: JavaMap,
+ msg_check: Optional[str] = JavaGatewayDefault.RESULT_MESSAGE_SUCCESS
+) -> Any:
+ if result[JavaGatewayDefault.RESULT_STATUS_KEYWORD].toString() != \
+ JavaGatewayDefault.RESULT_STATUS_SUCCESS:
+ raise RuntimeError(f"Failed when try to got result for java gateway")
+ if msg_check is not None and result[JavaGatewayDefault.RESULT_MESSAGE_KEYWORD] != msg_check:
+ raise ValueError(f"Get result state not success.")
+ return result
+
+# try to add decorators bu failed
+# def gateway_result_checker(func, *args, **kwargs):
Review comment:
Good catch! I forgot about this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie edited a comment on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie edited a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930982071
Questions: could I add `py4j.jar` to file `known-dependencies.txt` directly? for passing `check-LICENSE` CI? py4j license in https://github.com/bartdag/py4j/blob/master/LICENSE.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie edited a comment on pull request #6269: [WIP] Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie edited a comment on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-930200515
> Could I code style check and auto fix in my local env?
I found out we have check-style file and idea plugin in https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: [WIP] Init DS python SDK pydolphinscheduler
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-926672914
@caishunfeng FYI, I add you as this PR co-author manually in commit https://github.com/apache/dolphinscheduler/pull/6269/commits/9da1811b15670460c54513840e60391e1594c9f3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953469055
SonarCloud Quality Gate failed. 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6269&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6269&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953462577
> You need to also add the new dependency license to https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-dist/release-docs/LICENSE
I add it to `release-docs/LICENSE` now in 2da0790, thinks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953590596
PTAL @dailidong @lenboo @CalvinKirs @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#issuecomment-953586971
> When I run python define code, I could see workflow was created and process instance start but failed. The screenshot as below 
>
> And I see some error log look like
>
> ```
> 2021-10-28 11:17:07.160 INFO 30083 --- [tp905404580-123] o.a.d.a.service.impl.LoggerServiceImpl : log host : 192.168.1.102 , logPath : , logServer port : 50051
> 2021-10-28 11:17:07.160 INFO 30083 --- [tp905404580-123] o.a.d.service.log.LogClientService : roll view log, host : 192.168.1.102, port : 50051, path , skipLineNum 0 ,limit 1000
> 2021-10-28 11:17:07.193 WARN 30083 --- [tp905404580-123] o.a.d.remote.NettyRemotingClient : connect to Host{address='192.168.1.102:50051', ip='192.168.1.102', port=50051} error
> ```
After I inspect it, I find it cause due to my laptop cpu load too high and work do not available and task mark failed. Maybe we should add some notice for this?
But for this PR, I think is ready to go now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a change in pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on a change in pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#discussion_r737962985
##########
File path: dolphinscheduler-python/pom.xml
##########
@@ -0,0 +1,61 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+<modelVersion>4.0.0</modelVersion>
+<parent>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler</artifactId>
+ <version>2.0.0-SNAPSHOT</version>
+</parent>
+<artifactId>dolphinscheduler-python</artifactId>
+<name>${project.artifactId}</name>
+<packaging>jar</packaging>
+
+<dependencies>
+ <!-- dolphinscheduler -->
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-api</artifactId>
+ </dependency>
+
+ <!--springboot-->
+ <dependency>
+ <groupId>org.springframework.boot</groupId>
+ <artifactId>spring-boot-starter-web</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.springframework.boot</groupId>
+ <artifactId>spring-boot-starter-tomcat</artifactId>
+ </exclusion>
+ <exclusion>
+ <artifactId>log4j-to-slf4j</artifactId>
+ <groupId>org.apache.logging.log4j</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>net.sf.py4j</groupId>
+ <artifactId>py4j</artifactId>
+ <version>0.10.9</version>
Review comment:
the dependency should put in parent pom.xml to manage version easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a change in pull request #6269: Add Python API implementation of workflows-as-code
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on a change in pull request #6269:
URL: https://github.com/apache/dolphinscheduler/pull/6269#discussion_r737963794
##########
File path: dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/java_gateway.py
##########
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Optional
+
+from py4j.java_collections import JavaMap
+from py4j.java_gateway import JavaGateway, GatewayParameters
+
+from pydolphinscheduler.constants import JavaGatewayDefault
+
+
+def launch_gateway() -> JavaGateway:
+ # TODO Note that automatic conversion makes calling Java methods slightly less efficient because
+ # in the worst case, Py4J needs to go through all registered converters for all parameters.
+ # This is why automatic conversion is disabled by default.
+ gateway = JavaGateway(gateway_parameters=GatewayParameters(auto_convert=True))
+ return gateway
+
+
+def gateway_result_checker(
+ result: JavaMap,
+ msg_check: Optional[str] = JavaGatewayDefault.RESULT_MESSAGE_SUCCESS
+) -> Any:
+ if result[JavaGatewayDefault.RESULT_STATUS_KEYWORD].toString() != \
+ JavaGatewayDefault.RESULT_STATUS_SUCCESS:
+ raise RuntimeError(f"Failed when try to got result for java gateway")
+ if msg_check is not None and result[JavaGatewayDefault.RESULT_MESSAGE_KEYWORD] != msg_check:
+ raise ValueError(f"Get result state not success.")
+ return result
+
+# try to add decorators bu failed
+# def gateway_result_checker(func, *args, **kwargs):
Review comment:
Should the code remove?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org