You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "s0nskar (via GitHub)" <gi...@apache.org> on 2026/06/05 07:36:07 UTC
[PR] [CELEBORN-2349] Support worker supplied tags during startup and TagsManager hardening [celeborn]
s0nskar opened a new pull request, #3719:
URL: https://github.com/apache/celeborn/pull/3719
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
This PR lets a worker advertise tags at startup and makes tags part of WorkerInfo, so they will use existing Raft replication and worker-lifecycle cleanup. This also gel well with current implementation, allowing tags from config service to be merged with worker supplied tags.
Backward Compatible:
- The behaviour remains unchanged when no worker supplied tags are configured.
- A master-side config `celeborn.tags.worker.registration.enabled` (default: true), will allow to ignore worker-supplied tags.
- If worker is running on older version, worker supplied tags will be empty.
### Why are the changes needed?
`TagsManager` today only works off the dynamic-config store. There is no way for a worker to declare its own tags. This makes this feature a little hard to use.
After this PR a worker can supply its own tags and manage its own tags lifecycle and still allow tags overriding via config service. This will make worker tags feature a lot easier to use.
### Does this PR resolve a correctness bug?
<!-- Check if yes. The `correctness` label will be added/removed automatically. -->
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
<!-- Check if yes. -->
- [X] Yes
### How was this patch tested?
- Added Unit Tests.
- Working on end-to-end testing in local setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r4322640945
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/tags/TagsManager.scala:
##########
@@ -18,113 +18,70 @@
package org.apache.celeborn.service.deploy.master.tags
import java.util
-import java.util.{Collections, Set => JSet}
-import java.util.concurrent.ConcurrentHashMap
+import java.util.{Set => JSet}
+import java.util.concurrent.atomic.AtomicReference
import java.util.function.Predicate
import java.util.stream.Collectors
-import scala.collection.JavaConverters.{asScalaIteratorConverter, mapAsScalaConcurrentMapConverter}
+import scala.collection.JavaConverters._
import org.apache.celeborn.common.identity.UserIdentifier
import org.apache.celeborn.common.internal.Logging
import org.apache.celeborn.common.meta.WorkerInfo
-import org.apache.celeborn.common.util.JavaUtils
import org.apache.celeborn.server.common.service.config.ConfigService
class TagsManager(configService: Option[ConfigService]) extends Logging {
- private val defaultTagStore = JavaUtils.newConcurrentHashMap[String, JSet[String]]()
- private val addNewTagFunc =
- new util.function.Function[String, ConcurrentHashMap.KeySetView[String, java.lang.Boolean]]() {
- override def apply(t: String): ConcurrentHashMap.KeySetView[String, java.lang.Boolean] =
- ConcurrentHashMap.newKeySet[String]()
- }
+ private val tagStoreRef: AtomicReference[Map[String, JSet[String]]] =
+ new AtomicReference(Map.empty)
- private def getTagStore: ConcurrentHashMap[String, JSet[String]] = {
- configService match {
- case Some(cs) =>
- // TODO: Make configStore.getTags return ConcurrentMap
- JavaUtils.newConcurrentHashMap(cs.getSystemConfigFromCache.getTags)
- case _ =>
- defaultTagStore
- }
- }
+ private def updateSnapshot(): Unit =
+ tagStoreRef.set(configService.flatMap(cs =>
+ Option(cs.getSystemConfigFromCache.getTags)
+ .map(_.asScala.toMap)).getOrElse(Map.empty))
+
+ updateSnapshot()
+ configService.foreach(_.registerListenerOnConfigUpdate(() => updateSnapshot()))
+
+ private def tagStore: Map[String, JSet[String]] = tagStoreRef.get()
+
+ private def resolveTagsExpr(userIdentifier: UserIdentifier, clientTagsExpr: String): String =
+ configService.map { cs =>
+ val tagsMeta = cs
+ .getTenantUserConfigFromCache(userIdentifier.tenantId, userIdentifier.name)
+ .getWorkerTagsMeta
+ if (tagsMeta.preferClientTagExpr) clientTagsExpr else tagsMeta.tagsExpr
+ }.getOrElse(clientTagsExpr)
def getTaggedWorkers(
userIdentifier: UserIdentifier,
clientTagsExpr: String,
workers: util.List[WorkerInfo]): util.List[WorkerInfo] = {
- val tagsExpr = configService.flatMap { cs =>
- val config = cs.getTenantUserConfigFromCache(userIdentifier.tenantId, userIdentifier.name)
- val tagsMeta = config.getWorkerTagsMeta
- if (tagsMeta.preferClientTagExpr) {
- Some(clientTagsExpr)
- } else {
- Some(tagsMeta.tagsExpr)
- }
- }.getOrElse(clientTagsExpr)
+ val tags = resolveTagsExpr(userIdentifier, clientTagsExpr)
+ .split(',').map(_.trim).filter(_.nonEmpty)
- if (tagsExpr.isEmpty) {
- logWarning("No tags provided")
+ if (tags.isEmpty) {
+ logDebug("No tags provided, returning all workers")
return workers
}
- val tags = tagsExpr.split(",").map(_.trim)
-
- var workersForTags: Option[JSet[String]] = None
- tags.foreach { tag =>
- val taggedWorkers = getTagStore.getOrDefault(tag, Collections.emptySet())
- workersForTags match {
- case Some(w) =>
- w.retainAll(taggedWorkers)
- case _ =>
- workersForTags = Some(new util.HashSet[String](taggedWorkers))
- }
- }
-
- if (workersForTags.isEmpty) {
- logWarning(s"No workers for tags: $tagsExpr found in cluster")
- return Collections.emptyList()
- }
-
+ val store = tagStore
val workerTagsPredicate = new Predicate[WorkerInfo] {
- override def test(w: WorkerInfo): Boolean = workersForTags.get.contains(w.toUniqueId)
+ override def test(w: WorkerInfo): Boolean = tags.forall { tag =>
+ w.tags.contains(tag) ||
+ store.get(tag).exists(_.contains(w.toUniqueId))
+ }
Review Comment:
@s0nskar, [P2] Evaluate all requested tags against one consistent set of worker tags.
`w.tags` is read again for every tag, while `updateRegisterWorkerMeta` replaces that field on the existing `WorkerInfo` during re-registration. If a worker changes from `{a}` to `{b}` between the two checks, the expression `a,b` returns true even though neither registration contains both tags. This can select a worker that does not satisfy the complete tag expression.
I reproduced this deterministically on b67538acaadba9e3ff81485cdc9e84334ac05c9c: pause the reader after checking `contains(a)` on the old set, re-register the worker with `{b}` through the production metadata update path, then resume selection. The assertion that `a,b` selects no workers fails; all five existing `TagsManagerSuite` tests pass.
Please keep published tag sets unchanged, safely publish replacements (e.g. with volatile/atomic semantics), and capture the reference once per worker before evaluating `forall`. Please also add a regression test for this concurrent re-registration and selection case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3879143565
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -6901,6 +6903,24 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val TAGS_WORKER_REGISTRATION_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.tags.worker.registration.enabled")
+ .categories("master")
+ .version("0.7.0")
Review Comment:
Addressed
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -6901,6 +6903,24 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val TAGS_WORKER_REGISTRATION_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.tags.worker.registration.enabled")
+ .categories("master")
+ .version("0.7.0")
+ .doc("When true, the master honors tags advertised by workers at registration " +
+ "(merged with the config-store tags). When false, worker-supplied tags are ignored.")
+ .booleanConf
+ .createWithDefault(true)
+
+ val WORKER_TAGS: ConfigEntry[Seq[String]] =
+ buildConf("celeborn.worker.tags")
+ .categories("worker")
+ .version("0.7.0")
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-5474099356
@SteNicholas updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-4632134526
@SteNicholas can you please trigger Copilot magic on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-4659781549
>Tag changes don't refresh on a re-register over a live entry. AbstractMetaManager.updateRegisterWorkerMeta uses workersMap.putIfAbsent(...), so a worker that re-registers while its entry still exists keeps the old WorkerInfo (old/empty tags). This is pre-existing behavior shared with disk info, not introduced here — but combined with this feature it means new celeborn.worker.tags only take effect after the stale entry is evicted (worker-lost), not on an immediate restart/re-register. Worth a doc note or follow-up.
@SteNicholas What do you think about this? Should we update the tags in next heartbeat similar to disk infos. Or should we change the code slightly and update these values during re-registration itself.
I think it will be cleaner if replace the existing entry with new entry and update it's reference in availableWorkers.
```
synchronized (workersMap) {
availableWorkers.remove(workerInfo); // evict old entry which contains reference
workersMap.put(workerInfo.toUniqueId(), workerInfo);
...
updateAvailableWorkers(workerInfo); // re-adds new reference
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3879148651
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/tags/TagsManager.scala:
##########
@@ -19,112 +19,64 @@ package org.apache.celeborn.service.deploy.master.tags
import java.util
import java.util.{Collections, Set => JSet}
-import java.util.concurrent.ConcurrentHashMap
import java.util.function.Predicate
import java.util.stream.Collectors
-import scala.collection.JavaConverters.{asScalaIteratorConverter, mapAsScalaConcurrentMapConverter}
+import scala.collection.JavaConverters._
import org.apache.celeborn.common.identity.UserIdentifier
import org.apache.celeborn.common.internal.Logging
import org.apache.celeborn.common.meta.WorkerInfo
-import org.apache.celeborn.common.util.JavaUtils
import org.apache.celeborn.server.common.service.config.ConfigService
class TagsManager(configService: Option[ConfigService]) extends Logging {
- private val defaultTagStore = JavaUtils.newConcurrentHashMap[String, JSet[String]]()
- private val addNewTagFunc =
- new util.function.Function[String, ConcurrentHashMap.KeySetView[String, java.lang.Boolean]]() {
- override def apply(t: String): ConcurrentHashMap.KeySetView[String, java.lang.Boolean] =
- ConcurrentHashMap.newKeySet[String]()
- }
-
- private def getTagStore: ConcurrentHashMap[String, JSet[String]] = {
+ private def tagStore: Option[util.Map[String, JSet[String]]] = {
configService match {
- case Some(cs) =>
- // TODO: Make configStore.getTags return ConcurrentMap
- JavaUtils.newConcurrentHashMap(cs.getSystemConfigFromCache.getTags)
- case _ =>
- defaultTagStore
+ case Some(cs) => Option(cs.getSystemConfigFromCache.getTags)
+ case _ => None
}
}
Review Comment:
Addressed by making a snapshot of tagStore and using the listener support of configService.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "strelok89 (via GitHub)" <gi...@apache.org>.
strelok89 commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-5857707750
its not stale waiting for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-5112976842
@SteNicholas Updated the PR to address above comments, can you PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3371159432
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/tags/TagsManager.scala:
##########
@@ -19,112 +19,63 @@ package org.apache.celeborn.service.deploy.master.tags
import java.util
import java.util.{Collections, Set => JSet}
-import java.util.concurrent.ConcurrentHashMap
import java.util.function.Predicate
import java.util.stream.Collectors
-import scala.collection.JavaConverters.{asScalaIteratorConverter, mapAsScalaConcurrentMapConverter}
+import scala.collection.JavaConverters._
import org.apache.celeborn.common.identity.UserIdentifier
import org.apache.celeborn.common.internal.Logging
import org.apache.celeborn.common.meta.WorkerInfo
-import org.apache.celeborn.common.util.JavaUtils
import org.apache.celeborn.server.common.service.config.ConfigService
class TagsManager(configService: Option[ConfigService]) extends Logging {
- private val defaultTagStore = JavaUtils.newConcurrentHashMap[String, JSet[String]]()
- private val addNewTagFunc =
- new util.function.Function[String, ConcurrentHashMap.KeySetView[String, java.lang.Boolean]]() {
- override def apply(t: String): ConcurrentHashMap.KeySetView[String, java.lang.Boolean] =
- ConcurrentHashMap.newKeySet[String]()
- }
-
- private def getTagStore: ConcurrentHashMap[String, JSet[String]] = {
+ private def tagStore: Option[util.Map[String, JSet[String]]] = {
configService match {
- case Some(cs) =>
- // TODO: Make configStore.getTags return ConcurrentMap
- JavaUtils.newConcurrentHashMap(cs.getSystemConfigFromCache.getTags)
- case _ =>
- defaultTagStore
+ case Some(cs) => Option(cs.getSystemConfigFromCache.getTags)
+ case _ => None
}
}
+ private def resolveTagsExpr(userIdentifier: UserIdentifier, clientTagsExpr: String): String =
+ configService.map { cs =>
+ val tagsMeta = cs
+ .getTenantUserConfigFromCache(userIdentifier.tenantId, userIdentifier.name)
+ .getWorkerTagsMeta
+ if (tagsMeta.preferClientTagExpr) clientTagsExpr else tagsMeta.tagsExpr
+ }.getOrElse(clientTagsExpr)
+
def getTaggedWorkers(
userIdentifier: UserIdentifier,
clientTagsExpr: String,
workers: util.List[WorkerInfo]): util.List[WorkerInfo] = {
- val tagsExpr = configService.flatMap { cs =>
- val config = cs.getTenantUserConfigFromCache(userIdentifier.tenantId, userIdentifier.name)
- val tagsMeta = config.getWorkerTagsMeta
- if (tagsMeta.preferClientTagExpr) {
- Some(clientTagsExpr)
- } else {
- Some(tagsMeta.tagsExpr)
- }
- }.getOrElse(clientTagsExpr)
+ val tags = resolveTagsExpr(userIdentifier, clientTagsExpr)
+ .split(",").map(_.trim).filter(_.nonEmpty)
- if (tagsExpr.isEmpty) {
- logWarning("No tags provided")
+ if (tags.isEmpty) {
+ logDebug("No tags provided, returning all workers")
return workers
}
- val tags = tagsExpr.split(",").map(_.trim)
-
- var workersForTags: Option[JSet[String]] = None
- tags.foreach { tag =>
- val taggedWorkers = getTagStore.getOrDefault(tag, Collections.emptySet())
- workersForTags match {
- case Some(w) =>
- w.retainAll(taggedWorkers)
- case _ =>
- workersForTags = Some(new util.HashSet[String](taggedWorkers))
- }
- }
-
- if (workersForTags.isEmpty) {
- logWarning(s"No workers for tags: $tagsExpr found in cluster")
- return Collections.emptyList()
- }
-
val workerTagsPredicate = new Predicate[WorkerInfo] {
- override def test(w: WorkerInfo): Boolean = workersForTags.get.contains(w.toUniqueId)
+ override def test(w: WorkerInfo): Boolean = tags.forall { tag =>
+ w.tags.contains(tag) ||
+ tagStore.exists(_.getOrDefault(tag, Collections.emptySet()).contains(w.toUniqueId))
+ }
Review Comment:
`tagStore` is a method, and it’s invoked inside the per-worker/per-tag predicate. That re-fetches the config-service tags map for every `WorkerInfo` being tested (and for every tag in the expression), which is avoidable overhead on large clusters. Consider resolving the store map once per `getTaggedWorkers` call and closing over it in the predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3479554838
##########
common/src/main/proto/TransportMessages.proto:
##########
@@ -219,6 +221,7 @@ message PbMetaRegisterWorkerRequest {
map<string, PbResourceConsumption> userResourceConsumption = 7;
int32 internalPort = 8;
string networkLocation = 9;
+ repeated string tags = 12;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3479661622
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -370,6 +371,7 @@ public void updateRegisterWorkerMeta(
disks,
new HashMap<>());
workerInfo.lastHeartbeat_$eq(System.currentTimeMillis());
+ workerInfo.tags_$eq(new HashSet<>(tags));
Review Comment:
Agreed, updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3759730460
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -6901,6 +6903,24 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val TAGS_WORKER_REGISTRATION_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.tags.worker.registration.enabled")
+ .categories("master")
+ .version("0.7.0")
+ .doc("When true, the master honors tags advertised by workers at registration " +
+ "(merged with the config-store tags). When false, worker-supplied tags are ignored.")
+ .booleanConf
+ .createWithDefault(true)
+
+ val WORKER_TAGS: ConfigEntry[Seq[String]] =
+ buildConf("celeborn.worker.tags")
+ .categories("worker")
+ .version("0.7.0")
Review Comment:
```suggestion
.version("0.7.1")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3879145033
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/tags/TagsManager.scala:
##########
@@ -19,112 +19,64 @@ package org.apache.celeborn.service.deploy.master.tags
import java.util
import java.util.{Collections, Set => JSet}
-import java.util.concurrent.ConcurrentHashMap
import java.util.function.Predicate
import java.util.stream.Collectors
-import scala.collection.JavaConverters.{asScalaIteratorConverter, mapAsScalaConcurrentMapConverter}
+import scala.collection.JavaConverters._
import org.apache.celeborn.common.identity.UserIdentifier
import org.apache.celeborn.common.internal.Logging
import org.apache.celeborn.common.meta.WorkerInfo
-import org.apache.celeborn.common.util.JavaUtils
import org.apache.celeborn.server.common.service.config.ConfigService
class TagsManager(configService: Option[ConfigService]) extends Logging {
- private val defaultTagStore = JavaUtils.newConcurrentHashMap[String, JSet[String]]()
- private val addNewTagFunc =
- new util.function.Function[String, ConcurrentHashMap.KeySetView[String, java.lang.Boolean]]() {
- override def apply(t: String): ConcurrentHashMap.KeySetView[String, java.lang.Boolean] =
- ConcurrentHashMap.newKeySet[String]()
- }
-
- private def getTagStore: ConcurrentHashMap[String, JSet[String]] = {
+ private def tagStore: Option[util.Map[String, JSet[String]]] = {
configService match {
- case Some(cs) =>
- // TODO: Make configStore.getTags return ConcurrentMap
- JavaUtils.newConcurrentHashMap(cs.getSystemConfigFromCache.getTags)
- case _ =>
- defaultTagStore
+ case Some(cs) => Option(cs.getSystemConfigFromCache.getTags)
+ case _ => None
}
}
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-5810790621
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@celeborn.apache.org>.
github-actions[bot] commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-7004122555
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@celeborn.apache.org>.
github-actions[bot] closed pull request #3719: [CELEBORN-2349] Support worker supplied tags and TagsManager hardening
URL: https://github.com/apache/celeborn/pull/3719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-5928773706
@SteNicholas ping for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-5255759926
@s0nskar, please rebase the latest main branch for stable CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3760030698
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/tags/TagsManager.scala:
##########
@@ -19,112 +19,64 @@ package org.apache.celeborn.service.deploy.master.tags
import java.util
import java.util.{Collections, Set => JSet}
-import java.util.concurrent.ConcurrentHashMap
import java.util.function.Predicate
import java.util.stream.Collectors
-import scala.collection.JavaConverters.{asScalaIteratorConverter, mapAsScalaConcurrentMapConverter}
+import scala.collection.JavaConverters._
import org.apache.celeborn.common.identity.UserIdentifier
import org.apache.celeborn.common.internal.Logging
import org.apache.celeborn.common.meta.WorkerInfo
-import org.apache.celeborn.common.util.JavaUtils
import org.apache.celeborn.server.common.service.config.ConfigService
class TagsManager(configService: Option[ConfigService]) extends Logging {
- private val defaultTagStore = JavaUtils.newConcurrentHashMap[String, JSet[String]]()
- private val addNewTagFunc =
- new util.function.Function[String, ConcurrentHashMap.KeySetView[String, java.lang.Boolean]]() {
- override def apply(t: String): ConcurrentHashMap.KeySetView[String, java.lang.Boolean] =
- ConcurrentHashMap.newKeySet[String]()
- }
-
- private def getTagStore: ConcurrentHashMap[String, JSet[String]] = {
+ private def tagStore: Option[util.Map[String, JSet[String]]] = {
configService match {
- case Some(cs) =>
- // TODO: Make configStore.getTags return ConcurrentMap
- JavaUtils.newConcurrentHashMap(cs.getSystemConfigFromCache.getTags)
- case _ =>
- defaultTagStore
+ case Some(cs) => Option(cs.getSystemConfigFromCache.getTags)
+ case _ => None
}
}
Review Comment:
This changed from creating a defensive `ConcurrentHashMap` copy of `getTags` to directly using the map returned by the config cache. The removed code comment indicated the underlying map is not guaranteed to be concurrent; if that map is mutated during refresh, concurrent reads in `getTaggedWorkers/getTagsForWorker/getTagsForCluster` can observe torn state or throw (e.g., `ConcurrentModificationException`). Recommend restoring a thread-safe snapshot/copy per refresh (or having the config cache expose an immutable/concurrent view) before iterating/reading from it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@celeborn.apache.org>.
github-actions[bot] commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-6735638428
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-4807108212
@SteNicholas Updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3759850467
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/tags/TagsManager.scala:
##########
@@ -19,112 +19,64 @@ package org.apache.celeborn.service.deploy.master.tags
import java.util
import java.util.{Collections, Set => JSet}
-import java.util.concurrent.ConcurrentHashMap
import java.util.function.Predicate
import java.util.stream.Collectors
-import scala.collection.JavaConverters.{asScalaIteratorConverter, mapAsScalaConcurrentMapConverter}
+import scala.collection.JavaConverters._
import org.apache.celeborn.common.identity.UserIdentifier
import org.apache.celeborn.common.internal.Logging
import org.apache.celeborn.common.meta.WorkerInfo
-import org.apache.celeborn.common.util.JavaUtils
import org.apache.celeborn.server.common.service.config.ConfigService
class TagsManager(configService: Option[ConfigService]) extends Logging {
- private val defaultTagStore = JavaUtils.newConcurrentHashMap[String, JSet[String]]()
- private val addNewTagFunc =
- new util.function.Function[String, ConcurrentHashMap.KeySetView[String, java.lang.Boolean]]() {
- override def apply(t: String): ConcurrentHashMap.KeySetView[String, java.lang.Boolean] =
- ConcurrentHashMap.newKeySet[String]()
- }
-
- private def getTagStore: ConcurrentHashMap[String, JSet[String]] = {
+ private def tagStore: Option[util.Map[String, JSet[String]]] = {
configService match {
- case Some(cs) =>
- // TODO: Make configStore.getTags return ConcurrentMap
- JavaUtils.newConcurrentHashMap(cs.getSystemConfigFromCache.getTags)
- case _ =>
- defaultTagStore
+ case Some(cs) => Option(cs.getSystemConfigFromCache.getTags)
+ case _ => None
}
}
Review Comment:
Previously this code defensively copied the config-store tags into a concurrent map; now it directly uses the underlying `getTags` map. If that underlying map (or its value sets) can be updated concurrently by the config service, `getTaggedWorkers`/`getTagsForWorker` can observe races or throw (e.g., `ConcurrentModificationException`). Suggestion (mandatory for thread safety): take a snapshot copy per call (e.g., into a new `HashMap`/`ConcurrentHashMap` with copied sets) or reintroduce the prior `JavaUtils.newConcurrentHashMap(...)` approach so reads are stable.
##########
master/src/test/scala/org/apache/celeborn/service/deploy/master/tags/TagsManagerSuite.scala:
##########
@@ -199,4 +130,37 @@ class TagsManagerSuite extends CelebornFunSuite {
assert(taggedWorkers.contains(WORKER3))
}
}
+
+ test("getTaggedWorkers matches workers tagged via either config store or self-registration") {
+ tagsManager = new TagsManager(Option(configService))
+ val selfTaggedWorker = workerWithTags("host4", TAG1)
+ val all = List(WORKER1, WORKER2, WORKER3, selfTaggedWorker).asJava
+
+ val tagged = tagsManager.getTaggedWorkers(user, TAG1, all)
+ assert(tagged.size == 3) // WORKER1, WORKER2 (config store) + selfTagged (self)
+ assert(tagged.contains(WORKER1))
+ assert(tagged.contains(WORKER2))
+ assert(!tagged.contains(WORKER3))
+ assert(tagged.contains(selfTaggedWorker))
+ }
+
+ test("getTaggedWorkers matches a worker whose tags span config store and self-registration") {
+ tagsManager = new TagsManager(Option(configService))
+ // host1 already tagged via config service
+ val selfTaggedWorker = workerWithTags("host1", TAG2)
+ val all = List(WORKER1, WORKER2, WORKER3, selfTaggedWorker).asJava
Review Comment:
This test case specifically relies on the worker being the same identity as the config-store entry ('host1 already tagged via config service'), but `workerWithTags(\"host1\", ...)` likely produces a different `toUniqueId` (due to different ports) than `WORKER1`. Adjust the test to ensure `selfTaggedWorker.toUniqueId == WORKER1.toUniqueId` (e.g., create `WorkerInfo` with `WORKER1`’s ports, then set `tags`).
##########
master/src/test/scala/org/apache/celeborn/service/deploy/master/tags/TagsManagerSuite.scala:
##########
@@ -38,127 +38,58 @@ class TagsManagerSuite extends CelebornFunSuite {
private val workers = List(WORKER1, WORKER2, WORKER3).asJava
private val user = UserIdentifier("tenant_01", "Jerry")
+ private var configService: ConfigService = _
override def beforeEach(): Unit = {
super.beforeEach()
DynamicConfigServiceFactory.reset()
- }
-
- test("test tags manager") {
- tagsManager = new TagsManager(Option(null))
-
- tagsManager.addTagToWorker(TAG1, WORKER1.toUniqueId)
- tagsManager.addTagToWorker(TAG1, WORKER2.toUniqueId)
-
- tagsManager.addTagToWorker(TAG2, WORKER2.toUniqueId)
- tagsManager.addTagToWorker(TAG2, WORKER3.toUniqueId)
- {
- val taggedWorkers = tagsManager.getTaggedWorkers(user, TAG1, workers)
- assert(taggedWorkers.size == 2)
- assert(taggedWorkers.contains(WORKER1))
- assert(taggedWorkers.contains(WORKER2))
- assert(!taggedWorkers.contains(WORKER3))
- }
+ val conf = new CelebornConf()
+ conf.set(CelebornConf.DYNAMIC_CONFIG_STORE_BACKEND, "FS")
+ conf.set(
+ CelebornConf.DYNAMIC_CONFIG_STORE_FS_PATH.key,
+ getTestResourceFile("dynamicConfig-tags.yaml").getPath)
+ configService = DynamicConfigServiceFactory.getConfigService(conf)
+ }
- {
- val taggedWorkers = tagsManager.getTaggedWorkers(user, TAG2, workers)
- assert(taggedWorkers.size == 2)
- assert(!taggedWorkers.contains(WORKER1))
- assert(taggedWorkers.contains(WORKER2))
- assert(taggedWorkers.contains(WORKER3))
- }
+ private def workerWithTags(host: String, tags: String*): WorkerInfo = {
+ val w = new WorkerInfo(host, 111, 112, 113, 114, 115)
+ w.tags = new java.util.HashSet[String](tags.asJava)
+ w
+ }
Review Comment:
`workerWithTags` hard-codes ports, but tag lookup against the config-store is done via `worker.toUniqueId`. In tests that intend to merge config-store tags with self-registration for an existing worker, creating a `WorkerInfo` with the same host but different ports will likely produce a different `toUniqueId`, so the config-store tags won’t apply. Fix by constructing the tagged worker using the exact same host/port tuple as the target worker (e.g., derive ports from `WORKER1`/`WORKER2`), or mutate `tags` on an existing `WorkerInfo` instance used in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3759719894
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -6901,6 +6903,24 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val TAGS_WORKER_REGISTRATION_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.tags.worker.registration.enabled")
+ .categories("master")
+ .version("0.7.0")
Review Comment:
```suggestion
.version("0.7.1")
```
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -6901,6 +6903,24 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val TAGS_WORKER_REGISTRATION_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.tags.worker.registration.enabled")
+ .categories("master")
+ .version("0.7.0")
+ .doc("When true, the master honors tags advertised by workers at registration " +
+ "(merged with the config-store tags). When false, worker-supplied tags are ignored.")
+ .booleanConf
+ .createWithDefault(true)
+
+ val WORKER_TAGS: ConfigEntry[Seq[String]] =
+ buildConf("celeborn.worker.tags")
+ .categories("worker")
+ .version("0.7.0")
Review Comment:
```suggestion
.version("0.7.1")
```
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -6901,6 +6903,24 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val TAGS_WORKER_REGISTRATION_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.tags.worker.registration.enabled")
+ .categories("master")
+ .version("0.7.0")
Review Comment:
```suggestion
.version("0.7.1")
```
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -6901,6 +6903,24 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val TAGS_WORKER_REGISTRATION_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.tags.worker.registration.enabled")
+ .categories("master")
+ .version("0.7.0")
+ .doc("When true, the master honors tags advertised by workers at registration " +
+ "(merged with the config-store tags). When false, worker-supplied tags are ignored.")
+ .booleanConf
+ .createWithDefault(true)
+
+ val WORKER_TAGS: ConfigEntry[Seq[String]] =
+ buildConf("celeborn.worker.tags")
+ .categories("worker")
+ .version("0.7.0")
Review Comment:
```suggestion
.version("0.7.1")
```
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -6901,6 +6903,24 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val TAGS_WORKER_REGISTRATION_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.tags.worker.registration.enabled")
+ .categories("master")
+ .version("0.7.0")
Review Comment:
```suggestion
.version("0.7.1")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas closed pull request #3719: [CELEBORN-2349] Support worker supplied tags and TagsManager hardening
URL: https://github.com/apache/celeborn/pull/3719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r4258580097
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -505,6 +505,9 @@ private[celeborn] class Master(
.toMap.asJava
val userResourceConsumption =
PbSerDeUtils.fromPbUserResourceConsumption(pbRegisterWorker.getUserResourceConsumptionMap)
+ val tags =
+ if (conf.tagsWorkerRegistrationEnabled) pbRegisterWorker.getTagsList.asScala.toSet
+ else Set.empty[String]
Review Comment:
**[P2] Honor the disable flag for restored worker tags**
This check only covers incoming registrations. If workers register with tags while the flag is enabled, restarting an HA master with `celeborn.tags.worker.registration.enabled=false` still restores those tags from snapshots or Raft log replay. The restored workers remain registered through heartbeats, so they never pass through this check again, and `TagsManager.getTaggedWorkers` continues matching their self-tags despite the setting being disabled.
Please also gate the use of `WorkerInfo.tags` during worker selection so the current master's setting applies to restored/replayed state. I reproduced this by registering a tagged worker, saving the metadata snapshot, restoring it with the flag disabled, sending a heartbeat, and filtering by its self-tag: the worker is still selected. All five existing `TagsManagerSuite` tests passed; the added regression test failed as described.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-4901711200
@SteNicholas Ping for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-5102288359
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-4728100981
@SteNicholas I have addressed the review comments and added a concerns as well for the second point. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r3457745821
##########
common/src/main/proto/TransportMessages.proto:
##########
@@ -219,6 +221,7 @@ message PbMetaRegisterWorkerRequest {
map<string, PbResourceConsumption> userResourceConsumption = 7;
int32 internalPort = 8;
string networkLocation = 9;
+ repeated string tags = 12;
Review Comment:
`tags = 12` skips field numbers 10 and 11 here — this message only goes up to `networkLocation = 9`, so it leaves a confusing gap (looks copy-pasted from `PbRegisterWorker` above, where `12` is correct). `PbMetaRegisterWorkerRequest` also has no Java/Scala callers: it's only embedded in the proto (`registerWorkerRequest = 17`), and the master register path actually flows through `Resource.proto`'s `RegisterWorkerRequest` (which correctly got `tags = 10`). So this edit is functionally dead — I'd drop it, or use `tags = 10` if you keep it for symmetry.
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -370,6 +371,7 @@ public void updateRegisterWorkerMeta(
disks,
new HashMap<>());
workerInfo.lastHeartbeat_$eq(System.currentTimeMillis());
+ workerInfo.tags_$eq(new HashSet<>(tags));
Review Comment:
Anchoring the re-register refresh discussion here. On an immediate re-register over a still-live entry, the `putIfAbsent` below keeps the old `WorkerInfo`, so these fresh tags are silently dropped until the stale entry is evicted (worker-lost). Smallest fix that mirrors the heartbeat's in-place model and avoids the collateral of a full `put`:
```java
WorkerInfo existing = workersMap.putIfAbsent(workerInfo.toUniqueId(), workerInfo);
if (existing != null) {
existing.tags_$eq(new HashSet<>(tags));
}
```
No `availableWorkers` remove/add needed — it already holds the live object, and tags don't affect availability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#discussion_r4258580097
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -505,6 +505,9 @@ private[celeborn] class Master(
.toMap.asJava
val userResourceConsumption =
PbSerDeUtils.fromPbUserResourceConsumption(pbRegisterWorker.getUserResourceConsumptionMap)
+ val tags =
+ if (conf.tagsWorkerRegistrationEnabled) pbRegisterWorker.getTagsList.asScala.toSet
+ else Set.empty[String]
Review Comment:
@s0nskar, **Honor the disable flag for restored worker tags**. This check only covers incoming registrations. If workers register with tags while the flag is enabled, restarting an HA master with `celeborn.tags.worker.registration.enabled=false` still restores those tags from snapshots or Raft log replay. The restored workers remain registered through heartbeats, so they never pass through this check again, and `TagsManager.getTaggedWorkers` continues matching their self-tags despite the setting being disabled.
Please also gate the use of `WorkerInfo.tags` during worker selection so the current master's setting applies to restored/replayed state. I reproduced this by registering a tagged worker, saving the metadata snapshot, restoring it with the flag disabled, sending a heartbeat, and filtering by its self-tag: the worker is still selected. All five existing `TagsManagerSuite` tests passed; the added regression test failed as described.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2349] Support worker supplied tags and TagsManager hardening [celeborn]
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on PR #3719:
URL: https://github.com/apache/celeborn/pull/3719#issuecomment-5537600707
@SteNicholas PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org