You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/10 14:16:49 UTC
[GitHub] [dolphinscheduler] lhcg opened a new issue, #11404: [Bug] [UI] 存储未启用
lhcg opened a new issue, #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
What happened
I set values.xml
`sharedStoragePersistence:
enabled: true
mountPath: "/opt/soft"
accessModes:
- "ReadWriteMany"
## storageClassName must support the access mode: ReadWriteMany
storageClassName: "ds-shared-nfs"
storage: "2Gi"`
`fsFileResourcePersistence:
enabled: true
accessModes:
- "ReadWriteMany"
## storageClassName must support the access mode: ReadWriteMany
storageClassName: "ds-file-nfs"
storage: "2Gi"`
`DATA_BASEDIR_PATH: "/tmp/dolphinscheduler"
RESOURCE_STORAGE_TYPE: "HDFS"
RESOURCE_UPLOAD_PATH: "/dolphinscheduler"
FS_DEFAULT_FS: "file:///"`
### What you expected to happen
It's show "存储未启用" when I upload the spark-examples_2.11-2.4.7.jar.
### How to reproduce
storageClassName: "ds-shared-nfs"
storageClassName: "ds-file-nfs"
all bound
It's use the nfs.
### Anything else
root@dolphinscheduler-api-dbcf78666-72wff:/opt/dolphinscheduler/conf# pwd
/opt/dolphinscheduler/conf
root@dolphinscheduler-api-dbcf78666-72wff:/opt/dolphinscheduler/conf# cat common.properties
`data.basedir.path=/tmp/dolphinscheduler
# resource storage type: HDFS, S3, NONE
resource.storage.type=NONE
# resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resource.upload.path=/dolphinscheduler
# whether to startup kerberos
hadoop.security.authentication.startup.state=false
# java.security.krb5.conf path
java.security.krb5.conf.path=/opt/krb5.conf
# login user from keytab username
login.user.keytab.username=hdfs-mycluster@ESZ.COM
# login user from keytab path
login.user.keytab.path=/opt/hdfs.headless.keytab
# kerberos expire time, the unit is hour
kerberos.expire.time=2
# resource view suffixs
#resource.view.suffixs=txt,log,sh,bat,conf,cfg,py,java,sql,xml,hql,properties,json,yml,yaml,ini,js
# if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path
hdfs.root.user=hdfs
# if resource.storage.type=S3, the value like: s3a://dolphinscheduler; if resource.storage.type=HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir
fs.defaultFS=file:///`
why?
<img width="928" alt="image" src="https://user-images.githubusercontent.com/8449870/183924289-13152b13-0b8e-4b19-8bb4-9456cbb9c934.png">
### Version
3.0.0
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] geekyouth commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
geekyouth commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1368667152
common.properties
```
resource.hdfs.fs.defaultFS=hdfs://hadoop01:8020
```
not work
😣😣😣
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jubull commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by "jubull (via GitHub)" <gi...@apache.org>.
jubull commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1420167489
这个问题怎么处理
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yutianaiqingtian commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
yutianaiqingtian commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1211961140
I deploy the 3.0.0-release by helm and take place the same issue when I try to upload the file to File Manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yutianaiqingtian commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
yutianaiqingtian commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1219211775
@
> @yutianaiqingtian so actually I found out the difference between v2.0.6 and v3.0.0 - in v2.0.6 when starting the container there's a `startup-init-conf.sh` and it will replace all conf properties with env variables, thus common.configmap variables will be loaded into properties, see
>
> https://github.com/apache/dolphinscheduler/blob/419f18f8b8f28cea89210a6135ed48d723cb0941/docker/build/startup-init-conf.sh#L134
>
> However, this behavior is lost in v3.0.0
@aspexdaniel Great. waiting for you fix it and build new image. It seem the dockerfile that not included in the dev branch currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS closed issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
SbloodyS closed issue #11404: [Bug] [UI] Storage not enabled
URL: https://github.com/apache/dolphinscheduler/issues/11404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] KyleZhang0536 commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
KyleZhang0536 commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1211478109
Does it mean I have to modify common.properties in 3.0.0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1210761148
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
What happened
I set values.xml
`sharedStoragePersistence:
enabled: true
mountPath: "/opt/soft"
accessModes:
- "ReadWriteMany"
## storageClassName must support the access mode: ReadWriteMany
storageClassName: "ds-shared-nfs"
storage: "2Gi"`
`fsFileResourcePersistence:
enabled: true
accessModes:
- "ReadWriteMany"
## storageClassName must support the access mode: ReadWriteMany
storageClassName: "ds-file-nfs"
storage: "2Gi"`
`DATA_BASEDIR_PATH: "/tmp/dolphinscheduler"
RESOURCE_STORAGE_TYPE: "HDFS"
RESOURCE_UPLOAD_PATH: "/dolphinscheduler"
FS_DEFAULT_FS: "file:///"`
### What you expected to happen
It's show "存储未启用" when I upload the spark-examples_2.11-2.4.7.jar.
### How to reproduce
storageClassName: "ds-shared-nfs"
storageClassName: "ds-file-nfs"
all bound
It's use the nfs.
### Anything else
root@dolphinscheduler-api-dbcf78666-72wff:/opt/dolphinscheduler/conf# pwd
/opt/dolphinscheduler/conf
root@dolphinscheduler-api-dbcf78666-72wff:/opt/dolphinscheduler/conf# cat common.properties
`data.basedir.path=/tmp/dolphinscheduler
# resource storage type: HDFS, S3, NONE
resource.storage.type=NONE
# resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resource.upload.path=/dolphinscheduler
# whether to startup kerberos
hadoop.security.authentication.startup.state=false
# java.security.krb5.conf path
java.security.krb5.conf.path=/opt/krb5.conf
# login user from keytab username
login.user.keytab.username=hdfs-mycluster@ESZ.COM
# login user from keytab path
login.user.keytab.path=/opt/hdfs.headless.keytab
# kerberos expire time, the unit is hour
kerberos.expire.time=2
# resource view suffixs
#resource.view.suffixs=txt,log,sh,bat,conf,cfg,py,java,sql,xml,hql,properties,json,yml,yaml,ini,js
# if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path
hdfs.root.user=hdfs
# if resource.storage.type=S3, the value like: s3a://dolphinscheduler; if resource.storage.type=HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir
fs.defaultFS=file:///`
why?
<img width="928" alt="image" src="https://user-images.githubusercontent.com/8449870/183924289-13152b13-0b8e-4b19-8bb4-9456cbb9c934.png">
### Version
3.0.0
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dmzzzzhlj90 commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
dmzzzzhlj90 commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1212725831
I had the same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] aspexdaniel commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
aspexdaniel commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1216045940
I believe a broader range of config issues will be tracked by @kezhenxu94 at https://github.com/apache/dolphinscheduler/tree/k8s/config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] justindongnz commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
justindongnz commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1256889854
I had the same issue and tried to compare with v2.
And figured out that the behavior of startup process is different between these two versions.
V2.0.6 uses tpl to generate configuration files under path /opt/dolphinscheduler/conf/ but not in V3
To solve the problem, I used the work around to get it through by using a configmap inject file /opt/dolphinscheduler/conf/common.properties
Change the line:
resource.storage.type=NONE
to:
resource.storage.type=HDFS
fs.defaultFS=hdfs://mycluster:8020
to:
fs.defaultFS=file:///
Hope this would help you guys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] aspexdaniel commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
aspexdaniel commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1212833056
same problem with configuration with s3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] thj999 commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
thj999 commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1256074517
use 3.0.1, I had the same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Amazingzzl commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
Amazingzzl commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1298073217
I have the same problem, how can I solve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1210761448
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] aspexdaniel commented on issue #11404: [Bug] [UI] Storage not enabled
Posted by GitBox <gi...@apache.org>.
aspexdaniel commented on issue #11404:
URL: https://github.com/apache/dolphinscheduler/issues/11404#issuecomment-1214512504
@yutianaiqingtian so actually I found out the difference between v2.0.6 and v3.0.0 - in v2.0.6 when starting the container there's a `startup-init-conf.sh` where it will replace all conf properties with env variables, thus common.configmap variables will be loaded into properties, see https://github.com/apache/dolphinscheduler/blob/419f18f8b8f28cea89210a6135ed48d723cb0941/docker/build/startup-init-conf.sh
However, this behavior is lost in v3.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org