You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "GWphua (via GitHub)" <gi...@apache.org> on 2025/02/21 09:38:12 UTC
[PR] Peon alias (druid)
GWphua opened a new pull request, #17749:
URL: https://github.com/apache/druid/pull/17749
### Description
This PR aims to fix an issue introduced by #17738.
When we have task pods running in multiple namespaces, it will be difficult to tell which task pods are being started by which cluster, especially when we have very similar job names.
#### Added an alias tag to allow Kubernetes operators to identify which cluster the task pods are created from.
Here are some of the things I have considered when implementing the change:
1. The job name must respect the 63-character limit.
2. We can easily differentiate between tasks and namespaces.
3. The job name must be unique.
4. What if users are running everything within one namespace? Maybe users will still prefer the old way of displaying task ID.
Intuition:
`k8sJobName` is currently broken down into 2 segments: {taskId}-{hash(taskId)}. The hashing will take up 32 characters, and is needed to maintain uniqueness. Hence, we can only work with making changes to the `taskId` part. The first thought is to change `taskId` to namespace, so that we at least know which namespace a task pod belongs in.
However, I encountered cases where namespace have long names, and similar prefix? (Think `data-infra-druid-cluster...-ns1`, `data-infra-druid-cluster...-ns2`). So, I introduced a new `druid.indexer.runner.alias` config to allow users to provide an alias instead. Now, `k8sJobName` will be of the form {alias}-{hash(taskId)}. If alias is not provided, we will default to using the current {taskId}-{hash(taskId)} naming convention.
Note that this solution will not allow us to differentiate between tasks. However, this is the best I could come up with to work around the headache where there are hundreds of tasks with roughly the same job name.
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
You can now name your K8s job names using `druid.indexer.runner.alias`
<hr>
##### Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on PR #17749:
URL: https://github.com/apache/druid/pull/17749#issuecomment-2676133382
the name `druid.indexer.runner.alias` is not intuitive. Maybe `druid.indexer.runner.k8sTaskNamePrefix` ?
and please add document for this new configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #17749:
URL: https://github.com/apache/druid/pull/17749#issuecomment-2709644314
hi @FrankChen021, I will work on documentation and changes after #17738 is merged. Since this PR will not make sense without it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #17749:
URL: https://github.com/apache/druid/pull/17749#issuecomment-2783599925
> Note that this solution will not allow us to differentiate between tasks in Kubernetes. However, we can still see our tasks and their corresponding taskID via the Web Console.
Note: My approach involves changing the method of K8sTaskId, prompting change in a lot other classes. This may not be the best way, please advise if there's a more elegant solution.
Yeah, @GWphua , I am not sure if this is a real improvement over what we had earlier.
With a prefix specified, all task pods launched by a cluster will end up looking the same (except for the hash part, which is not exactly human readable). It would be difficult to identify the pod for any given Druid task ID.
Plus, requiring users to specify a prefix just to be able to tell pods apart seems undesirable to me.
That said, using the substring of the task name was not all that great either.
I am not sure what a better alternative would be, but will try to give it some more thought.
Side note:
IIUC from the changes, this shouldn't have any rolling upgrade implications, i.e. task pods launched by an Overlord where the `k8sTaskPodNamePrefix` was not specified can still be detected by an Overlord where this config has been specified (or an Overlord where the value of the config has been updated).
@GWphua , could you please confirm if this is actually the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #17749:
URL: https://github.com/apache/druid/pull/17749#issuecomment-2791514659
> Currently I think we can add some text to the doc to give a guidance on changing of this property which requires users to stop all running tasks first.
Yes, let's update the docs and we should also clearly call this out in the release notes and upgrade notes for this feature (just adding it in the PR description should suffice).
cc: @GWphua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 merged PR #17749:
URL: https://github.com/apache/druid/pull/17749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #17749:
URL: https://github.com/apache/druid/pull/17749#issuecomment-2771237431
> the name `druid.indexer.runner.alias` is not intuitive. Maybe `druid.indexer.runner.k8sTaskNamePrefix` ? and please add document for this new configuration.
Hi @FrankChen021, I have changed the name to `druid.indexer.runner.k8sTaskPodNamePrefix`. Also, added the documentation for this new configuration. Feel free to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on PR #17749:
URL: https://github.com/apache/druid/pull/17749#issuecomment-2790131653
This is the same as the other newly introduced `overlordNamespace`, which requires users to configure them proply: users should have a clear pic that how his clusters are going to deploy.
Currently I think we can add some text to the doc to give a guidance on changing of this property which requires users to stop all running tasks first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #17749:
URL: https://github.com/apache/druid/pull/17749#discussion_r2027325124
##########
extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunnerConfig.java:
##########
@@ -371,6 +382,12 @@ public Builder withOverlordNamespace(String overlordNamespace)
return this;
}
+ public Builder withk8sTaskPodNamePrefix(String k8sTaskPodNamePrefix)
Review Comment:
```suggestion
public Builder withK8sTaskPodNamePrefix(String k8sTaskPodNamePrefix)
```
suggested by Copilot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #17749:
URL: https://github.com/apache/druid/pull/17749#discussion_r2027317372
##########
docs/development/extensions-core/k8s-jobs.md:
##########
@@ -745,6 +745,17 @@ If you are running multiple Druid clusters and would like to have a dedicated na
Druid will tag Kubernetes jobs with a `druid.overlord.namespace` label. This label helps Druid filter out Kubernetes jobs belonging to other namespaces.
+##### Differentiating Task Pods Created From Multiple Namespaces
+
+When we have task pods running in multiple namespaces, it will be difficult to tell which task pods are being started by which cluster. You can specify a task name prefix, `druid.indexer.runner.k8sTaskPodNamePrefix`, to apply your specified prefix to all task pods created by your cluster.
Review Comment:
```suggestion
When we have task pods started by overlords of different Druid clusters runnng in different K8S namespaces, it will be difficult to tell which task pods are being started by which overlord or Druid cluster. You can specify a task name prefix, `druid.indexer.runner.k8sTaskPodNamePrefix`, to apply your specified prefix to all task pods created by your cluster.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #17749:
URL: https://github.com/apache/druid/pull/17749#issuecomment-2785573505
> IIUC from the changes, this shouldn't have any rolling upgrade implications, i.e. task pods launched by an Overlord where the k8sTaskPodNamePrefix was not specified can still be detected by an Overlord where this config has been specified (or an Overlord where the value of the config has been updated).
@GWphua , could you please confirm if this is actually the case.
Hi @kfaraz, unfortunately there will be disruptions to rolling upgrades.
The old tasks (without configured `k8sTaskPodNamePrefix`) will not be detected, and Overlord will restart the on-going tasks with the configured `k8sTaskPodNamePrefix`. From the user's perspective, the web console will show the old tasks as FAILED.
On our side, we need to notify our users to expect this to happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Allow users to differentiate task pods from different namespaces in multi-namespace settings (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #17749:
URL: https://github.com/apache/druid/pull/17749#issuecomment-2790056386
> The old tasks (without configured k8sTaskPodNamePrefix) will not be detected, and Overlord will restart the on-going tasks with the configured k8sTaskPodNamePrefix. From the user's perspective, the web console will show the old tasks as FAILED.
Oh, this is problematic, @GWphua , as the Overlord is losing track of those tasks and re-launching them, even though the original tasks are still running and will finish normally. This will lead to duplicate data and possibly metadata inconsistency errors in case of streaming ingestion.
Is there any way to mitigate this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org