You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/01 09:55:03 UTC
[GitHub] [dolphinscheduler] ocean-zhc opened a new issue, #11236: [Bug] [DataX] DataX Variable configuration problem
ocean-zhc opened a new issue, #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
k8s deployment：
ds-version:3.0.0-beat2
<img width="1828" alt="image" src="https://user-images.githubusercontent.com/46189785/182122211-6ddc4250-cd41-4196-9a1c-abe8844e8bba.png">
<img width="2579" alt="image" src="https://user-images.githubusercontent.com/46189785/182122241-75fbb38a-4bc1-46a7-b656-557ceb77e439.png">
datax.py script cannot read!
### What you expected to happen
datax.py script available
### How to reproduce
1、https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/kubernetes.html
2、
```
kubectl cp -n sjjcb-ds datax.tar.gz dolphinscheduler-worker-0:/opt/soft
```
3、
```
kubectl exec -n sjjcb-ds -it dolphinscheduler-worker-0 bash
cd /opt/soft
tar -zxvf datax.tar.gz
rm -rf datax.tar.gz
```
4、
```
export DATAX_HOME=/opt/soft/datax
export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH
```
<img width="1711" alt="image" src="https://user-images.githubusercontent.com/46189785/182123163-fa9635c2-5231-428d-a747-559a2a4a7584.png">
5、Execute task execution, prompt the above error
### Anything else
_No response_
### Version
3.0.0-beta-2
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simsicon commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
simsicon commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1238797242
> > By the way, 3.0.0 docker image does not contain python2 by default, I think it's also an issue? @kezhenxu94
Right, I have to built a docker image with python 2.0 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1262226551
i agree with `source`, but may this is not root cause for this case. I faced this case many times, the root casue is some file is not exists, for example: cannot find /bin/datax.py, if we can fix this file, `source: not found` also will be fixed.
Please check https://github.com/apache/dolphinscheduler/discussions/11290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simsicon commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
simsicon commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1222170228
@ocean-zhc could you please share the solution you end up with?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simsicon commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
simsicon commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1260363621
> I am experiencing the same issue with 3.0.0, it seems this line is not able to read the env properly, I have tested it has nothing to do with the env settings. `https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java#L96`
> > By the way, 3.0.0 docker image does not contain python2 by default, I think it's also an issue? @kezhenxu94
>
> <img alt="image" width="1042" src="https://user-images.githubusercontent.com/15833811/188648449-628caaeb-a3ff-4ba1-862b-4a0e06cb63dd.png">
>
> the root cause is `source: not found`, if `source` command can't execute normally, the env file will not work
`source` is not available in `/bin/sh`, it's replacement in `/bin/sh` is `.`, I think you can either use `/bin/bash` instead of `/bin/sh` or use `.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] davidzollo commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
davidzollo commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1238157752
> I am experiencing the same issue with 3.0.0, it seems this line is not able to read the env properly, I have tested it has nothing to do with the env settings. `https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java#L96`
>
> By the way, 3.0.0 docker image does not contain python2 by default, I think it's also an issue? @kezhenxu94
the root cause is `source: not found`, if `source` command can't execute normally, the env file will not work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simsicon commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
simsicon commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1222195643
I am experiencing the same issue with 3.0.0, it seems this line is not able to read the env properly, I have tested it has nothing to do with the env settings.
`https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java#L96`
By the way, 3.0.0 docker image does not contain python2 by default, I think it's also an issue? @kezhenxu94
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] wittychen commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
wittychen commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1342807796
> > > i agree with `source`, but may this is not root cause for this case. I faced this case many times, the root casue is some file is not exists, for example: cannot find /bin/datax.py, if we can fix this file, `source: not found` also will be fixed. Please check #11290
> >
> >
> > I have already submit this pr #12180 to solve the issue, for your reference
>
> Great! thanks simsicon
I see that version 3.0.3 has been repaired, but the image warehouse does not have the image of the Linux/amd version. Can you upload one?
https://hub.docker.com/layers/apache/dolphinscheduler-worker/3.0.3/images/sha256-9f408b93dbe2387e61d8d3458b46fe6a72b9b654f0376c1c1ccf8c9502271ccd?context=explore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1200981698
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1263045903
> > i agree with `source`, but may this is not root cause for this case. I faced this case many times, the root casue is some file is not exists, for example: cannot find /bin/datax.py, if we can fix this file, `source: not found` also will be fixed. Please check #11290
>
> I have already submit this pr #12180 to solve the issue, for your reference
Great! thanks simsicon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ocean-zhc closed issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
ocean-zhc closed issue #11236: [Bug] [DataX] DataX Variable configuration problem
URL: https://github.com/apache/dolphinscheduler/issues/11236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simsicon commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
simsicon commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1262364466
> i agree with `source`, but may this is not root cause for this case. I faced this case many times, the root casue is some file is not exists, for example: cannot find /bin/datax.py, if we can fix this file, `source: not found` also will be fixed. Please check #11290
I have already submit this pr to solve the issue, for your reference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simsicon commented on issue #11236: [Bug] [DataX] DataX Variable configuration problem
Posted by GitBox <gi...@apache.org>.
simsicon commented on issue #11236:
URL: https://github.com/apache/dolphinscheduler/issues/11236#issuecomment-1260364049
I just tested it with `3.0.1`, this is still a issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org