You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/09/29 10:05:54 UTC
[GitHub] [incubator-dolphinscheduler] yefei-ddyw opened a new issue #3854: What the purpose of queue about DolphinScheduler?
yefei-ddyw opened a new issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854
*For better global communication, please give priority to using English description, thx! *
*Please review https://dolphinscheduler.apache.org/en-us/docs/development/issue.html when describe an issue.*
**Describe the question**
What the purpose of queue about DolphinScheduler?
**Which version of DolphinScheduler:**
-[1.3.2-preview]
**Additional context**
I have some questions about the queue of ds.The document shows the queue of ds is due to MapReduce,Spark and so on.But,I find that the logs of yarn(host:19888) shows the queue is tenant code of ds,not the queue of ds.Why ?
And the document shows the tenant code is the user of Linux.So,the queue of yarn is the tenant code?
Therefore,I test the tenant and queue of ds.Here are the results.
**Requirement or improvement**
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw removed a comment on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw removed a comment on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-700605622
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] geosmart commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
geosmart commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-701111010
1. config queue in ds, queue name is yarn queue name;
2. tenant config queue;
3. define dag, config tenant ,so the ds will submit task with tenant's confied queue in yarn cluster;
---
the ds_queue just a name for display,the yarn queue is the actualy working queue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw edited a comment on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw edited a comment on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-707519042
>
>
> if you have more questions about this question, please reopen it
我的DS是1.3.2，CDH是6.2.1。我可以确定，DS所有的队列，CDH的yarn里都有。无论我怎么测试，最后yarn日志里显示的队列都是DS里设置的租户编码。我的yarn的放置规则如下：
**1.仅当池 已在运行时制定 存在时使用该池.
2.Use the pool root..[username] only if the pool and root. both exists.(下拉项中的池名称是hdfs，root.hdfs是作为父池)
3.使用池 root. 。
日规则始终满足。不会使用后续规则。**
DS里租户编码跟租户名称一样，队列名称与队列值是一样。我在DS里设置了**root.hdfs.mapreduce，root.hdfs.spark，root.hdfs.flink这三个队列**，所有的租户、用户关联的队列都是这三个之一。在yarn的资源池里设置了6个不同的资源池，包括了DS的一样的队列root.hdfs.mapreduce、root.hdfs.spark、root.hdfs.flink，还包括了三个不同的root.hdfs.DS租户编码。结果yarn日志里显示的队列还是root.hdfs.DS租户编码。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw removed a comment on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw removed a comment on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-700605982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-707519042
>
>
> if you have more questions about this question, please reopen it
我的DS是1.3.2，CDH是6.2.1。我可以确定，DS所有的队列，CDH的yarn里都有。无论我怎么测试，最后yarn日志里显示的队列都是DS里设置的租户编码。我的yarn的放置规则如下：
**1.仅当池 已在运行时制定 存在时使用该池.
2.Use the pool root..[username] only if the pool and root. both exists.(下拉项中的池名称是hdfs，root.hdfs是作为父池)
3.使用池 root. 。
日规则始终满足。不会使用后续规则。**
DS里租户编码跟租户名称一样，队列名称与队列值是一样。我在DS里设置了**root.hdfs.mapreduce，root.hdfs.spark，root.hdfs.flink这三个队列**，所有的租户、用户关联的队列都是这三个之一。在yarn的资源池里设置了6个不同的资源池，包括了DS的一样的队列root.hdfs.mapreduce、root.hdfs.spark、root.hdfs.flink，还包括了三个不同的root.hdfs.DS租户编码。结果yarn日志里显示的队列还是root.hdfs.DS租户编码。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw removed a comment on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw removed a comment on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-700606635
[user_tenant_queue_test.xlsx](https://github.com/apache/incubator-dolphinscheduler/files/5298177/user_tenant_queue_test.xlsx)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-700605982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-700606635
[user_tenant_queue_test.xlsx](https://github.com/apache/incubator-dolphinscheduler/files/5298177/user_tenant_queue_test.xlsx)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-706239875
if you have more questions about this question, please reopen it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw removed a comment on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw removed a comment on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-701119362
yarn_pool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong closed issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
dailidong closed issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw edited a comment on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw edited a comment on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-707519042
>
>
> if you have more questions about this question, please reopen it
我的DS是1.3.2，CDH是6.2.1。我可以确定，DS所有的队列，CDH的yarn里都有。无论我怎么测试，最后yarn日志里显示的队列都是DS里设置的租户编码。我的yarn的放置规则如下：
**1.仅当池 已在运行时制定 存在时使用该池.
2.Use the pool root..[username] only if the pool and root. both exists.(下拉项中的池名称是hdfs，root.hdfs是作为父池)
3.使用池 root. 。
日规则始终满足。不会使用后续规则。**
DS里租户编码跟租户名称一样，队列名称与队列值是一样。我在DS里设置了**root.hdfs.mapreduce，root.hdfs.spark，root.hdfs.flink这三个队列**，所有的租户、用户关联的队列都是这三个之一。在yarn的资源池里设置了6个不同的资源池，包括了DS的一样的队列root.hdfs.mapreduce、root.hdfs.spark、root.hdfs.flink，还包括了三个不同的root.hdfs.DS租户编码。结果yarn日志里显示的队列还是root.hdfs.DS租户编码。
![Uploading yarn运行日志.jpg…]()
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-700605622
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-706238861
you should use the queue name on the yarn cluster, as an example root.root.004 is yarn queue name,
then you can set queue name root.root.004 on DolphinScheduler, when DS submit task, it will submit the task to root.root.004
yarn queue.
so first you should confirm the queue name you used in DolphinScheduler exists on the yarn cluster, DS only submit the task to the corresponding yarn queue through the queue name
---
DolphinScheduler仅仅是提交任务到对应名称的yarn队列中去，请保持DS里的队列名字在yarn集群的队列列表里是存在的
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-701119362
yarn_pool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yefei-ddyw commented on issue #3854: What the purpose of queue about DolphinScheduler?
Posted by GitBox <gi...@apache.org>.
yefei-ddyw commented on issue #3854:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3854#issuecomment-701119037
> 1. config queue in ds, queue name is yarn queue name;
> 2. tenant config queue;
> 3. define dag, config tenant ,so the ds will submit task with tenant's confied queue in yarn cluster;
>
> the ds_queue just a name for display,the yarn queue is the actualy working queue
I think yarn_queue shows that if the name of ds_queue is default,the second name of yarn_queue is root;if the name of ds_queue is not default the second name of yarn_queue is the name of ds_queue.Because the ds_queue can be same ,so the third name of yarn_queue is the code of ds_code.You can test more confirm what I write.
The first name(root) of yarn_queue is the parent pool for all pools.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org