You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/10/13 21:03:59 UTC
[GitHub] [dolphinscheduler] Arturo-Penas-Rial opened a new issue, #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Arturo-Penas-Rial opened a new issue, #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
I am testing version 3.1 of DS, and the changes introduced are impressive. I would like to make a POC with DS to introduce it in my organization, but I need your help. I have tried to create an image from apache/dolphinscheduler-worker, but with no success.
I need help to have a "worker" node that has PENTAHO CARTE installed (in order to invoke the ETL process execution via HTTP). An example of an image with CARTE is "nentangso/pentaho-kettle-carte:latest"
Excuse me, but I need your help. I have tried to search for manuals, blogs, without success. I think DS is an excellent product (it should be more popular than Apache Airflow!), but I need CARTE integration. Can you help me, provide a model image, or provide documentation to achieve it?
### Use case
Users who use ETL from PENTAHO PDI, who want to use DS, need a worker node with PENTAHO PDI with its services exposed through CARTE (HTTP).
Especially useful to compete against Apache Airflow which has integrations with NiFi and PENTAHO. Would allow users with traditional ETL to use DS orchestration
### Related issues
https://github.com/apache/dolphinscheduler/pull/12366
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] William-GuoWei commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
William-GuoWei commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1285031809
Hi, I think here are some solutions for you:
1. DS worker is not like Airflow, it can support install Kettle Client on Worker, but not a docker of Kettle with DS client. So you need to have a kettle docker besides the worker server (DS Worker can run on the same server on DS Master).
2. You can easily use Http Task in DS to trigger Kettle job with http request.
3. I am thinking about add a Kettle task in DS, and that will be easier for you to use kettle as a docker to run ETL jobs and close it after ETL finish.
For now, If your ETL job is on cloud, I think maybe you need create 3 task for Kettle,
1. Create a shell or python task of start a docker with Kettle server
2. Create a Http task or shell task to trigger Kettle job with http request
3. Create a shell or python task of close the docker with Kettle server
I hope that can help you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1283517462
>The state of a task definition, named "Need fault tolerance", that’s mean? I have a deployment based on docker compose with only one worker node.
The `Need fault tolerance` is the state means that these task instances will rerun again by tolerance (the old one was finished and create a new task instance for same workflow instance).
This fault tolerance mechanism is triggered when the worker server down (task instances tolerance) or master server down(workflow instances tolerance).
see https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/architecture/design.html, the `Fault-Tolerant Design` part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1278234601
> ### Search before asking
>
> - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
>
>
> ### Description
>
> I am testing version 3.1 of DS, and the changes introduced are impressive. I would like to make a POC with DS to introduce it in my organization, but I need your help. I have tried to create an image from apache/dolphinscheduler-worker, but with no success.
>
>
>
> I need help to have a "worker" node that has PENTAHO CARTE installed (in order to invoke the ETL process execution via HTTP). An example of an image with CARTE is "nentangso/pentaho-kettle-carte:latest"
>
Just write your own Dockerfile and base from the apache/dolphinscheduler-worker, build and push to you registry and use it. This is a rather common sense to extend a Docker image just like how we build apache/dolphinscheduler-worker from openjdk base image.
>
> Excuse me, but I need your help. I have tried to search for manuals, blogs, without success. I think DS is an excellent product (it should be more popular than Apache Airflow!), but I need CARTE integration. Can you help me, provide a model image, or provide documentation to achieve it?
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Arturo-Penas-Rial commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
Arturo-Penas-Rial commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1282483201
The name of a workflow definition can change (modified)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Arturo-Penas-Rial commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
Arturo-Penas-Rial commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1282455994
Hi @zhongjiajie
Yes :-) still tying. Thanks for your response and availability. Now the news ...
- I reach to modify and activate a worker image including PENTAHO PDI (docker pull arturopenas/dolphinscheduler-pdi-worker:latest),
+ but I cannot access from outside the container. Can I reuse the default port 1235, to expose internal HTTP services?
+ I would like to check if the image customized is Ok, so ... is there any mechanism to check if the image is OK for DS?
- The state of a task definition, named "Need fault tolerance", that’s mean? I have a deployment based on docker compose with only one worker node.
I fell DS is a great product!, but it will be easier to adopt if a set of worker images with integrations like PENTAHO PDI, NIFI, would be available.
Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] nielifeng commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
nielifeng commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1283557168
Can you provide me with your e-mail or other contact information?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Arturo-Penas-Rial commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
Arturo-Penas-Rial commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1285695661
Hi @William-GuoWei ... Let me check if I understood :-)
In Airflow, the worker node can contain both PENTAHO PDI and CARTE, I managed to do it and it works. However, I would like to use DS, but I still can't get a worker node with PENTAHO PDI and CARTE.
What I am trying to do is to extend the DS worker image, to incorporate PENTAHO. Is it possible? I do not want to have a docker of Kettle with DS client inside a "worker".
From your previous message I understand that the above would not be possible, so I should create a docker node with PENTAHO PDI and CARTE (not integrating the DS orchestration) and access it through the worker? If so, how do I schedule a task (ETL) since the PENTAHO node would be out of scope?
My ETLs are in local server, and the idea was to migrate it to a DS "worker" node, sharing a directory with the ETL files and from the DS agenda invoke via HTTP the execution in CARTE of those ETLs.
I haven't found any communication to guide me on how to customize the "worker". It would be great if DS could have images with pre-installed typocos products like for example PENTAHO!
How do you think I can move forward?
Thanks and have a nice day!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Arturo-Penas-Rial commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
Arturo-Penas-Rial commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1282481781
One more :-) @zhongjiajie ...
When I try to execute a DAG the following error log occurs. I believe there is a miss configuration, can you guide me?
[LOG-PATH]: /opt/dolphinscheduler/logs/20221018/7222558634432_2-6-9.log, [HOST]: Host{address='[172.20.0.5:1234](http://172.20.0.5:1234/)', ip='172.20.0.5', port=1234}
[INFO] 2022-10-18 14:12:40.511 +0000 - Begin to pulling task
[INFO] 2022-10-18 14:12:40.525 +0000 - Begin to initialize task
[INFO] 2022-10-18 14:12:40.525 +0000 - Set task startTime: Tue Oct 18 14:12:40 UTC 2022
[INFO] 2022-10-18 14:12:40.525 +0000 - Set task envFile: /opt/dolphinscheduler/conf/dolphinscheduler_env.sh
[INFO] 2022-10-18 14:12:40.526 +0000 - Set task appId: 6_9
[INFO] 2022-10-18 14:12:40.526 +0000 - End initialize task
[INFO] 2022-10-18 14:12:40.527 +0000 - Set task status to TaskExecutionStatus{code=1, desc='running'}
[ERROR] 2022-10-18 14:12:40.601 +0000 - Task execute failed, due to meet an exception
org.apache.dolphinscheduler.plugin.task.api.TaskException: TenantCode: Inquilino #2 doesn't exist
at org.apache.dolphinscheduler.server.worker.utils.TaskExecutionCheckerUtils.checkTenantExist(TaskExecutionCheckerUtils.java:62)
at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.beforeExecute(WorkerTaskExecuteRunnable.java:186)
at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:149)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
[INFO] 2022-10-18 14:12:40.603 +0000 - Get a exception when execute the task, will send the task execute result to master, the current task execute result is TaskExecutionStatus{code=6, desc='failure'}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1282035897
Hi @Arturo-Penas-Rial are you still there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1283493942
And I wonder whether PENTAHO PDI has a client or CLI interface that we can submit our job and execute them? Actually, we only need to submit the job in DolphinScheduler worker, it can run this job in PENTAHO PDI itself
running an application with a web UI interface in DolphinScheduler worker maybe not be the best practice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Arturo-Penas-Rial commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
Arturo-Penas-Rial commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1285914076
Well I reach to extend the worker node image, also PENTAHO Carte is running on this node, but DS find as unhealty state ...:-( The worker node log does not show errors! Can you give me some guidance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1278177073
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Arturo-Penas-Rial commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
Arturo-Penas-Rial commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1283887143
Many thanks for your quick response!, I will follow your guidelines. My email is arturo.penas@gmail.com.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1278385638
and we have related docs about it, you can see https://dolphinscheduler.apache.org/en-us/docs/3.0.1/user_doc/contribute/development-environment-setup.html section "Docker image build", we have "create custom Docker images" about how to create custom images base on official release images
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1283488290
> I reach to modify and activate a worker image including PENTAHO PDI
can you share what is your Dockerfile for this PENTAHO PDI worker docker images?
> Can I reuse the default port 1235
You can not, 1235 is for workers to communicate with other servers, it is being used and will throw an error because is be used.
> so ... is there any mechanism to check if the image is OK for DS
you can check in tab "monitor" to see whether the worker is up, and then run a simple task to find our it work or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1283479225
> The name of a workflow definition can change (modified)?
It support, you can change the workflow name according to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #12367: Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #12367:
URL: https://github.com/apache/dolphinscheduler/issues/12367#issuecomment-1283527607
> One more :-) @zhongjiajie ...
>
> When I try to execute a DAG the following error log occurs. I believe there is a miss configuration, can you guide me?
>
> [LOG-PATH]: /opt/dolphinscheduler/logs/20221018/7222558634432_2-6-9.log, [HOST]: Host{address='[172.20.0.5:1234](http://172.20.0.5:1234/)', ip='172.20.0.5', port=1234} [INFO] 2022-10-18 14:12:40.511 +0000 - Begin to pulling task [INFO] 2022-10-18 14:12:40.525 +0000 - Begin to initialize task [INFO] 2022-10-18 14:12:40.525 +0000 - Set task startTime: Tue Oct 18 14:12:40 UTC 2022 [INFO] 2022-10-18 14:12:40.525 +0000 - Set task envFile: /opt/dolphinscheduler/conf/dolphinscheduler_env.sh [INFO] 2022-10-18 14:12:40.526 +0000 - Set task appId: 6_9 [INFO] 2022-10-18 14:12:40.526 +0000 - End initialize task [INFO] 2022-10-18 14:12:40.527 +0000 - Set task status to TaskExecutionStatus{code=1, desc='running'} [ERROR] 2022-10-18 14:12:40.601 +0000 - Task execute failed, due to meet an exception org.apache.dolphinscheduler.plugin.task.api.TaskException: TenantCode: Inquilino #2 doesn't exist at org.apache.dolphinscheduler.server.worker.utils.TaskExecutionCheckerUtils.checkTenantExist(TaskEx
ecutionCheckerUtils.java:62) at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.beforeExecute(WorkerTaskExecuteRunnable.java:186) at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:149) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) [INFO] 2022-10-18 14:12:40.603 +0000 - Get a exception when execute the task, will send the task
execute result to master, the current task execute result is TaskExecutionStatus{code=6, desc='failure'}
>
> Note: When I access "workflow definition" and I try to execute the DAG the following error message appear "master does not exist", but all process are up and running.
Hi @Arturo-Penas-Rial I try at local and it can create a new tenant by default config. I think you can try to execute cmd ` sudo useradd -g user-group new-user` in the worker node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org