You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/10/14 08:51:41 UTC
[GitHub] [incubator-dolphinscheduler] yefei-ddyw opened a new issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
yefei-ddyw opened a new issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910
*For better global communication, please give priority to using English description, thx! *
*Please review https://dolphinscheduler.apache.org/en-us/docs/development/issue.html when describe an issue.*
**Describe the question**
Why my pool of yarn couldn't shows the same and DS queue?
**Which version of DolphinScheduler:**
-[1.3.2-preview]
**Which version of CDH:**
6.2.1
**Additional context**
I have some questions about the queue of ds.The document shows the queue of ds is due to MapReduce,Spark and so on.But,I find that the logs of yarn(host:19888) shows the queue is tenant code of ds,not the queue of ds.Why ?
And the document shows the tenant code is the user of Linux.So,the queue of yarn is the tenant code?
Therefore,I test the tenant and queue of ds.Here are the results.
The following is the configuration of DS and Resources Pools of yarn.
The image is the queue of DS.
The image is the users of DS.
The image is the tenant of DS.
The image is all the pools of yarn.
The image is the resource pool placement rules of yarn.
The image is the resource pool placement rules of yarn also.
The image is the logs of yarn.
**Requirement or improvement**
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] xingchun-chen commented on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
xingchun-chen commented on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-749962134
who can look the problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-750125625
@yefei-ddyw @xingchun-chen
User | Tenant | User Queue (Submitted) | Tenant Queue (Ignored) | Yarn Queue (Actual)
-- | -- | -- | -- | --
1 | ddyw | root.hdfs.flink | root.hdfs.mapreduce | root.hdfs.ddyw
2 | dd011 | root.hdfs.mapreduce | root.hdfs.spark | root.hdfs.dd011
3 | ddyw1 | root.hdfs.spark | root.hdfs.flink | root.hdfs.ddyw1
Both user queue and tenant queue are inconsistent.
I guess the reason is as follows:
The tenant ddyw submits a job with user queue `root.hdfs.flink`, not the tenant queue `root.hdfs.mapreduce`, so that `root.hdfs.flink` doesn't take effect. It may be caused by queue permissions.
So the job doesn't meet rule 1 in Dynamic Resource Pool Configuration, so the second rule takes effect by using `root..[username]` with middle pool name `hdfs`.
In summary, the job submitted by the tenant ddyw will use `root.hdfs.ddyw` queue in yarn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-750125625
@yefei-ddyw @xingchun-chen
User | Tenant | User Queue (Submitted) | Tenant Queue (Ignored) | Yarn Queue (Actual)
-- | -- | -- | -- | --
1 | ddyw | root.hdfs.flink | root.hdfs.mapreduce | root.hdfs.ddyw
2 | dd011 | root.hdfs.mapreduce | root.hdfs.spark | root.hdfs.dd011
3 | ddyw1 | root.hdfs.spark | root.hdfs.flink | root.hdfs.ddyw1
Both tenant queue and user queue are inconsistent.
I guess the reason is as follows:
The tenant ddyw submits a job with user queue `root.hdfs.flink`, not the tenant queue `root.hdfs.mapreduce`, so that `root.hdfs.flink` doesn't take effect. It may be caused by queue permissions.
So the job doesn't meet rule 1 in Dynamic Resource Pool Configuration, so the second rule takes effect by using `root..[username]` with middle pool name `hdfs`.
In summary, the job submitted by the tenant ddyw will use `root.hdfs.ddyw` queue in yarn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-750125625
@yefei-ddyw @chenxingchun
User | Tenant | User Queue (Submitted) | Tenant Queue (Ignored) | Yarn Queue (Actual)
-- | -- | -- | -- | --
1 | ddyw | root.hdfs.flink | root.hdfs.mapreduce | root.hdfs.ddyw
2 | dd011 | root.hdfs.mapreduce | root.hdfs.spark | root.hdfs.dd011
3 | ddyw1 | root.hdfs.spark | root.hdfs.flink | root.hdfs.ddyw1
I guess the reason is as follows:
The tenant ddyw submits a job with user queue `root.hdfs.flink`, not the tenant queue `root.hdfs.mapreduce`, so that `root.hdfs.flink` doesn't take effect. It may be caused by queue permissions.
So the job doesn't meet rule 1 in Dynamic Resource Pool Configuration, so the second rule takes effect by using `root..[username]` with middle pool name `hdfs`.
In summary, the job submitted by the tenant ddyw will use `root.hdfs.ddyw` queue in yarn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-750125625
@yefei-ddyw @xingchun-chen
User | Tenant | User Queue (Submitted) | Tenant Queue (Ignored) | Yarn Queue (Actual)
-- | -- | -- | -- | --
1 | ddyw | root.hdfs.flink | root.hdfs.mapreduce | root.hdfs.ddyw
2 | dd011 | root.hdfs.mapreduce | root.hdfs.spark | root.hdfs.dd011
3 | ddyw1 | root.hdfs.spark | root.hdfs.flink | root.hdfs.ddyw1
Both user queue and tenant queue are inconsistent.
I guess the reason is as follows:
The tenant ddyw submits a job with user queue `root.hdfs.flink`, not the tenant queue `root.hdfs.mapreduce`, so that `root.hdfs.flink` doesn't take effect. It may be caused by queue permissions.
And then, the job doesn't meet rule 1 in Dynamic Resource Pool Configuration, so the second rule takes effect by using `root..[username]` with middle pool name `hdfs`.
In summary, the job submitted by the tenant ddyw will use `root.hdfs.ddyw` queue in yarn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-750041793
@yefei-ddyw
Priority of queue: queue from `other args` > user queue > tenant queue
Flink: 1.3.x has bug about queue, so that the job will be submitted to the `default` queue regardless of the existing user or tenant queue. The latest dev branch has fixed it.
Spark: No problem
MR: No problem
Could this information help you explain the current problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen commented on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen commented on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-750125625
User | Tenant | User Queue (Submitted) | Tenant Queue (Ignored) | Yarn Queue (Actual)
-- | -- | -- | -- | --
1 | ddyw | root.hdfs.flink | root.hdfs.mapreduce | root.hdfs.ddyw
2 | dd011 | root.hdfs.mapreduce | root.hdfs.spark | root.hdfs.dd011
3 | ddyw1 | root.hdfs.spark | root.hdfs.flink | root.hdfs.ddyw1
I guess the reason is as follows:
The tenant ddyw submits a job with user queue `root.hdfs.flink`, not the tenant queue `root.hdfs.mapreduce`, so that `root.hdfs.flink` doesn't take effect. It may be caused by queue permissions.
So the job doesn't meet rule 1 in Dynamic Resource Pool Configuration, so the second rule takes effect by using `root..[username]` with middle pool name `hdfs`.
In summary, the job submitted by the tenant ddyw will use `root.hdfs.ddyw` queue in yarn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen closed issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen closed issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen commented on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen commented on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-750041793
Priority of queue: queue from `other args` > user queue > tenant queue
Flink: 1.3.x has bug about queue, so that the job will be submitted to the `default` queue regardless of the existing user or tenant queue. The latest dev branch has fixed it.
Spark: No problem
MR: No problem
Can this information help you explain the current problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-750041793
@yefei-ddyw
Priority of queue: queue from `other args` > user queue > tenant queue
Flink: 1.3.x has bug about queue, so that the job will be submitted to the `default` queue regardless of the existing user or tenant queue. The latest dev branch has fixed it.
Spark: No problem
MR: No problem
Can this information help you explain the current problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen commented on issue #3910: Why my pool of yarn couldn't shows the same and DS queue?
Posted by GitBox <gi...@apache.org>.
chengshiwen commented on issue #3910:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3910#issuecomment-817113001
Close the question since it has been replied
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org