You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/07/08 23:55:01 UTC
[GitHub] [kafka] guozhangwang opened a new pull request, #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
guozhangwang opened a new pull request, #12397:
URL: https://github.com/apache/kafka/pull/12397
This PR is to consolidate https://github.com/apache/kafka/pull/12338 and https://github.com/apache/kafka/pull/12384
In order to integrate with the state updater, we would need to refactor the TaskManager and Task interfaces. This PR achieved the following purposes:
1. Separate active and standby tasks in the Tasks placeholder, plus adding pendingActiveTasks and pendingStandbyTasks into Tasks. The exposed active/standby tasks from the Tasks set would only be mutated by a single thread, and the pending tasks hold for those tasks that are assigned but cannot be actively managed yet. For now they include two scenarios: a) tasks from unknown sub-topologies and hence cannot be initialized, b) tasks that are pending for being recycled from active to standby and vice versa. Note case b) would be added in a follow-up PR.
2. Extract any logic that mutates a task out of the Tasks / TaskCreators. Tasks should only be a place for maintaining the set of tasks, but not for manipulations of a task; and TaskCreators should only be used for creating the tasks, but not for anything else. These logic are all migrated into TaskManger.
3. While doing 2) I noticed we have a couple of minor issues in the code where we duplicate the closing logics, so I also cleaned them up in the following way:
a) When closing a task, we first trigger the corresponding closeClean/Dirty function; then we remove the task from `Tasks` bookkeeping, and for active task we also remove its task producer if EOS-V1 is used.
b) For closing dirty, we swallow the exception from close call and the remove task producer call; for closing clean, we store the thrown exception from either close call or the remove task producer, and then rethrow at the end of the caller. The difference though is that, for the exception from close call we need to retry close it dirty; for the exception from the remove task producer we do not need to re-close it dirty.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r925880702
##########
clients/src/main/java/org/apache/kafka/common/utils/Utils.java:
##########
@@ -1307,9 +1307,9 @@ public Set<Characteristics> characteristics() {
}
@SafeVarargs
- public static <E> Set<E> union(final Supplier<Set<E>> constructor, final Set<E>... set) {
+ public static <E> Set<E> union(final Supplier<Set<E>> constructor, final Collection<E>... set) {
final Set<E> result = constructor.get();
- for (final Set<E> s : set) {
+ for (final Collection<E> s : set) {
Review Comment:
Good point, I also feel using `Set` is safer for de-dup effects. Will change it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
cadonna commented on PR #12397:
URL: https://github.com/apache/kafka/pull/12397#issuecomment-1191467025
Build failures are unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r925917621
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StandbyTask.java:
##########
@@ -234,6 +236,43 @@ public void closeCleanAndRecycleState() {
log.info("Closed clean and recycled state");
}
+ /**
+ * Create an active task from this standby task without closing and re-initializing the state stores.
+ * The task should have been in suspended state when calling this function
+ *
+ * TODO: we should be able to not need the input partitions as input param in future but always reuse
+ * the task's input partitions when we have fixed partitions -> tasks mapping
+ */
+ public StreamTask recycle(final Time time,
Review Comment:
Yeah, I was also wondering if we can simplify this chain of calls from `TaskManager` -> `Tasks` -> `StandbyTaskCreator`/`ActiveTaskCreator` -> `StreamTask`/`StandbyTask`. And I also thought it might be better to postpone this refactoring. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r925881686
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/AbstractTask.java:
##########
@@ -55,25 +57,28 @@ public abstract class AbstractTask implements Task {
protected Map<TopicPartition, Long> offsetSnapshotSinceLastFlush = null;
protected final TaskId id;
+ protected final TaskConfig config;
protected final ProcessorTopology topology;
protected final StateDirectory stateDirectory;
protected final ProcessorStateManager stateMgr;
+
private final long taskTimeoutMs;
AbstractTask(final TaskId id,
final ProcessorTopology topology,
final StateDirectory stateDirectory,
final ProcessorStateManager stateMgr,
final Set<TopicPartition> inputPartitions,
- final long taskTimeoutMs,
+ final TaskConfig config,
final String taskType,
final Class<? extends AbstractTask> clazz) {
this.id = id;
this.stateMgr = stateMgr;
this.topology = topology;
+ this.config = config;
this.inputPartitions = inputPartitions;
this.stateDirectory = stateDirectory;
- this.taskTimeoutMs = taskTimeoutMs;
+ this.taskTimeoutMs = config.taskTimeoutMs;
Review Comment:
Ack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r917191699
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -34,29 +33,40 @@
import java.util.TreeMap;
import java.util.stream.Collectors;
+import static org.apache.kafka.common.utils.Utils.filterMap;
+import static org.apache.kafka.common.utils.Utils.union;
+
+/**
+ * All tasks contained by the Streams instance.
+ *
+ * Note that these tasks are shared between the TaskManager (stream thread) and the StateUpdater (restore thread),
+ * i.e. all running active tasks are processed by the former and all restoring active tasks and standby tasks are
+ * processed by the latter.
+ */
class Tasks {
private final Logger log;
- private final TopologyMetadata topologyMetadata;
-
- private final Map<TaskId, Task> allTasksPerId = Collections.synchronizedSortedMap(new TreeMap<>());
- private final Map<TaskId, Task> readOnlyTasksPerId = Collections.unmodifiableMap(allTasksPerId);
- private final Collection<Task> readOnlyTasks = Collections.unmodifiableCollection(allTasksPerId.values());
// TODO: change type to `StreamTask`
private final Map<TaskId, Task> activeTasksPerId = new TreeMap<>();
+ // TODO: change type to `StandbyTask`
+ private final Map<TaskId, Task> standbyTasksPerId = new TreeMap<>();
+
+ // Tasks may have been assigned for a NamedTopology that is not yet known by this host. When that occurs we stash
+ // these unknown tasks until either the corresponding NamedTopology is added and we can create them at last, or
+ // we receive a new assignment and they are revoked from the thread.
+
+ // Tasks may have been assigned but not yet created because:
+ // 1. They are for a NamedTopology that is yet known by this host.
+ // 2. They are to be recycled from an existing restoring task yet to be returned from the task updater.
+ //
+ // When that occurs we stash these pending tasks until either they are finally clear to be created,
+ // or they are revoked from a new assignment.
+ private final Map<TaskId, Set<TopicPartition>> pendingActiveTasks = new HashMap<>();
Review Comment:
This is mainly for case 1) in the description.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -65,13 +75,10 @@ class Tasks {
private Consumer<byte[], byte[]> mainConsumer;
Tasks(final LogContext logContext,
- final TopologyMetadata topologyMetadata,
Review Comment:
This is for case 2) in description: we detect unknown named topologies in the caller (`TaskManager`) so `Tasks` would be just dumb book-keeping.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StandbyTaskCreator.java:
##########
@@ -73,30 +67,14 @@ class StandbyTaskCreator {
);
}
- void removeRevokedUnknownTasks(final Set<TaskId> assignedTasks) {
Review Comment:
This is for case 2) in description.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/AbstractTask.java:
##########
@@ -151,7 +156,8 @@ final void transitionTo(final Task.State newState) {
@Override
public void updateInputPartitions(final Set<TopicPartition> topicPartitions, final Map<String, List<String>> allTopologyNodesToSourceTopics) {
- this.inputPartitions = topicPartitions;
+ this.inputPartitions.clear();
Review Comment:
Minor cleanup: do deep copy since the passed in `topicPartition` param maybe modified.
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/DefaultStateUpdaterTest.java:
##########
@@ -83,7 +83,6 @@ class DefaultStateUpdaterTest {
private final Time time = new MockTime(1L);
private final StreamsConfig config = new StreamsConfig(configProps());
private final ChangelogReader changelogReader = mock(ChangelogReader.class);
- private final java.util.function.Consumer<Set<TopicPartition>> offsetResetter = topicPartitions -> { };
Review Comment:
This is a leftover from previous commit, we do not need this anymore.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -573,6 +572,45 @@ public void closeCleanAndRecycleState() {
log.info("Closed clean and recycled state");
}
+ /**
+ * Create a standby task from this active task without closing and re-initializing the state stores.
+ * The task should have been in suspended state when calling this function
+ *
+ * TODO: we should be able to not need the input partitions as input param in future but always reuse
+ * the task's input partitions when we have fixed partitions -> tasks mapping
+ */
+ public StandbyTask recycle(final Set<TopicPartition> inputPartitions) {
Review Comment:
Consolidate the logic of recycling within a task, so that it can be extracted from the creator later for recycling after tasks returned from state updater.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -346,22 +365,34 @@ public void handleAssignment(final Map<TaskId, Set<TopicPartition>> activeTasks,
throw first.getValue();
}
- tasks.handleNewAssignmentAndCreateTasks(activeTasksToCreate, standbyTasksToCreate, activeTasks.keySet(), standbyTasks.keySet());
+ tasks.createTasks(activeTasksToCreate, standbyTasksToCreate);
+ }
+
+ private Map<TaskId, Set<TopicPartition>> pendingTasksToCreate(final Map<TaskId, Set<TopicPartition>> tasksToCreate) {
+ final Map<TaskId, Set<TopicPartition>> pendingTasks = new HashMap<>();
+ final Iterator<Map.Entry<TaskId, Set<TopicPartition>>> iter = tasksToCreate.entrySet().iterator();
+ while (iter.hasNext()) {
+ final Map.Entry<TaskId, Set<TopicPartition>> entry = iter.next();
+ final TaskId taskId = entry.getKey();
+ if (taskId.topologyName() != null && !topologyMetadata.namedTopologiesView().contains(taskId.topologyName())) {
+ pendingTasks.put(taskId, entry.getValue());
+ iter.remove();
+ }
+ }
+ return pendingTasks;
}
- private void handleCloseAndRecycle(final Set<Task> tasksToRecycle,
- final Set<Task> tasksToCloseClean,
- final Set<Task> tasksToCloseDirty,
- final Map<TaskId, Set<TopicPartition>> activeTasksToCreate,
- final Map<TaskId, Set<TopicPartition>> standbyTasksToCreate,
- final LinkedHashMap<TaskId, RuntimeException> taskCloseExceptions) {
+ private void closeAndRecycleTasks(final Map<Task, Set<TopicPartition>> tasksToRecycle,
Review Comment:
This is a cleanup: we construct the pending tasks to recycle outside the handleCloseAndRecycle so that we do not need to pass in activeTasksToCreate / standbyTasksToCreate anymore.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -859,16 +899,6 @@ void closeAndCleanUpTasks(final Collection<Task> activeTasks, final Collection<T
closeTaskDirty(task);
}
- // TODO: change type to `StreamTask`
Review Comment:
This is for case 3) in the description: this is already in the `completeTaskCloseClean` so is a duplicated code.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/ActiveTaskCreator.java:
##########
@@ -142,33 +135,16 @@ StreamsProducer threadProducer() {
return threadProducer;
}
- void removeRevokedUnknownTasks(final Set<TaskId> assignedTasks) {
Review Comment:
This is for case 2) in description.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -285,34 +286,52 @@ public void handleAssignment(final Map<TaskId, Set<TopicPartition>> activeTasks,
final LinkedHashMap<TaskId, RuntimeException> taskCloseExceptions = new LinkedHashMap<>();
final Map<TaskId, Set<TopicPartition>> activeTasksToCreate = new HashMap<>(activeTasks);
final Map<TaskId, Set<TopicPartition>> standbyTasksToCreate = new HashMap<>(standbyTasks);
+ final Map<Task, Set<TopicPartition>> tasksToRecycle = new HashMap<>();
final Comparator<Task> byId = Comparator.comparing(Task::id);
- final Set<Task> tasksToRecycle = new TreeSet<>(byId);
final Set<Task> tasksToCloseClean = new TreeSet<>(byId);
final Set<Task> tasksToCloseDirty = new TreeSet<>(byId);
- // first rectify all existing tasks
+ tasks.purgePendingTasks(activeTasks.keySet(), standbyTasks.keySet());
+
+ // first rectify all existing tasks:
+ // 1. for tasks that are already owned, just update input partitions / resume and skip re-creating them
+ // 2. for tasks that have changed active/standby status, just recycle and skip re-creating them
+ // 3. otherwise, close them since they are no longer owned
for (final Task task : tasks.allTasks()) {
- if (activeTasks.containsKey(task.id()) && task.isActive()) {
- tasks.updateInputPartitionsAndResume(task, activeTasks.get(task.id()));
- activeTasksToCreate.remove(task.id());
- } else if (standbyTasks.containsKey(task.id()) && !task.isActive()) {
- tasks.updateInputPartitionsAndResume(task, standbyTasks.get(task.id()));
- standbyTasksToCreate.remove(task.id());
- } else if (activeTasks.containsKey(task.id()) || standbyTasks.containsKey(task.id())) {
- // check for tasks that were owned previously but have changed active/standby status
- tasksToRecycle.add(task);
+ final TaskId taskId = task.id();
+ if (activeTasksToCreate.containsKey(taskId)) {
+ if (task.isActive()) {
+ final Set<TopicPartition> topicPartitions = activeTasksToCreate.get(taskId);
Review Comment:
This is for case 2) in description: extract all `task.mutate` functions out of `Tasks` and only inside `TaskManager`.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/AbstractTask.java:
##########
@@ -106,7 +111,7 @@ public TaskId id() {
@Override
public Set<TopicPartition> inputPartitions() {
- return inputPartitions;
+ return Collections.unmodifiableSet(inputPartitions);
Review Comment:
Minor cleanup: do not allow callers to modify the set.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -292,7 +292,6 @@ public void suspend() {
partitionGroup.clear();
} finally {
transitToSuspend();
- log.info("Suspended running");
Review Comment:
Minor cleanup: duplicated logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r925892759
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StandbyTask.java:
##########
@@ -234,6 +236,43 @@ public void closeCleanAndRecycleState() {
log.info("Closed clean and recycled state");
}
+ /**
+ * Create an active task from this standby task without closing and re-initializing the state stores.
+ * The task should have been in suspended state when calling this function
+ *
+ * TODO: we should be able to not need the input partitions as input param in future but always reuse
+ * the task's input partitions when we have fixed partitions -> tasks mapping
+ */
+ public StreamTask recycle(final Time time,
Review Comment:
I re-think about this a bit, and I felt just having `recycle` in the task is not sufficient as a refactoring, since we'd need to still call 1) task.closeCleanAndRecycleState, 2) taskCreator.createXTaskFromY(), which would call 3) task.recycle() in order.
I think it's better to extract the creators out of the `Tasks` (then also extract the main consumer as well since it's only needed for the creation of the tasks) into `TaskManager`, and merge the above steps into a single function. But since this PR is getting complicated enough I'd probably defer this to another PR.
I could either revert this function or just keep it here without the test coverage, and then further consolidate the logic in a follow-up PR, which one do you prefer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna merged pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
cadonna merged PR #12397:
URL: https://github.com/apache/kafka/pull/12397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r927019010
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -262,51 +294,30 @@ Collection<Task> tasks(final Collection<TaskId> taskIds) {
// TODO: change return type to `StreamTask`
Collection<Task> activeTasks() {
- return readOnlyActiveTasks;
- }
-
- Collection<Task> allTasks() {
- return readOnlyTasks;
- }
-
- Collection<Task> notPausedActiveTasks() {
- return new ArrayList<>(readOnlyActiveTasks)
- .stream()
- .filter(t -> !topologyMetadata.isPaused(t.id().topologyName()))
- .collect(Collectors.toList());
+ return Collections.unmodifiableCollection(activeTasksPerId.values());
}
- Collection<Task> notPausedTasks() {
- return new ArrayList<>(readOnlyTasks)
- .stream()
- .filter(t -> !topologyMetadata.isPaused(t.id().topologyName()))
- .collect(Collectors.toList());
+ /**
+ * All tasks returned by any of the getters are read-only and should NOT be modified;
+ * and the returned task could be modified by other threads concurrently
+ */
+ Set<Task> allTasks() {
+ return union(HashSet::new, new HashSet<>(activeTasksPerId.values()), new HashSet<>(standbyTasksPerId.values()));
}
- Set<TaskId> activeTaskIds() {
- return readOnlyActiveTaskIds;
+ Set<TaskId> allTaskIds() {
+ return union(HashSet::new, activeTasksPerId.keySet(), standbyTasksPerId.keySet());
}
- Set<TaskId> standbyTaskIds() {
- return readOnlyStandbyTaskIds;
- }
-
- // TODO: change return type to `StreamTask`
- Map<TaskId, Task> activeTaskMap() {
- return readOnlyActiveTasksPerId;
- }
-
- // TODO: change return type to `StandbyTask`
- Map<TaskId, Task> standbyTaskMap() {
- return readOnlyStandbyTasksPerId;
- }
-
- Map<TaskId, Task> tasksPerId() {
- return readOnlyTasksPerId;
+ Map<TaskId, Task> allTasksPerId() {
+ final Map<TaskId, Task> ret = new HashMap<>();
+ ret.putAll(activeTasksPerId);
+ ret.putAll(standbyTasksPerId);
+ return ret;
Review Comment:
Sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r925471614
##########
clients/src/main/java/org/apache/kafka/common/utils/Utils.java:
##########
@@ -1307,9 +1307,9 @@ public Set<Characteristics> characteristics() {
}
@SafeVarargs
- public static <E> Set<E> union(final Supplier<Set<E>> constructor, final Set<E>... set) {
+ public static <E> Set<E> union(final Supplier<Set<E>> constructor, final Collection<E>... set) {
final Set<E> result = constructor.get();
- for (final Set<E> s : set) {
+ for (final Collection<E> s : set) {
Review Comment:
This seems a bit weird to me. I see that you changed this because `Tasks#allTaskIds()` returns a `Collection` instead of a `Set`. I think it is fine to let `Tasks#allTaskIds()` return a `Set`. The other reason you changed that is that is `Tasks#allTasks()` computing its result from two collections. Those two collections can be transformed to sets. Maybe we should also consider to change the return type of `allTasks()` to a set.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -34,29 +33,40 @@
import java.util.TreeMap;
import java.util.stream.Collectors;
+import static org.apache.kafka.common.utils.Utils.filterMap;
+import static org.apache.kafka.common.utils.Utils.union;
+
+/**
+ * All tasks contained by the Streams instance.
+ *
+ * Note that these tasks are shared between the TaskManager (stream thread) and the StateUpdater (restore thread),
+ * i.e. all running active tasks are processed by the former and all restoring active tasks and standby tasks are
+ * processed by the latter.
+ */
class Tasks {
private final Logger log;
- private final TopologyMetadata topologyMetadata;
-
- private final Map<TaskId, Task> allTasksPerId = Collections.synchronizedSortedMap(new TreeMap<>());
- private final Map<TaskId, Task> readOnlyTasksPerId = Collections.unmodifiableMap(allTasksPerId);
- private final Collection<Task> readOnlyTasks = Collections.unmodifiableCollection(allTasksPerId.values());
// TODO: change type to `StreamTask`
private final Map<TaskId, Task> activeTasksPerId = new TreeMap<>();
+ // TODO: change type to `StandbyTask`
+ private final Map<TaskId, Task> standbyTasksPerId = new TreeMap<>();
+
+ // Tasks may have been assigned for a NamedTopology that is not yet known by this host. When that occurs we stash
+ // these unknown tasks until either the corresponding NamedTopology is added and we can create them at last, or
+ // we receive a new assignment and they are revoked from the thread.
+
+ // Tasks may have been assigned but not yet created because:
+ // 1. They are for a NamedTopology that is yet known by this host.
+ // 2. They are to be recycled from an existing restoring task yet to be returned from the task updater.
Review Comment:
```suggestion
// 2. They are to be recycled from an existing restoring task yet to be returned from the state updater.
```
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/AbstractTask.java:
##########
@@ -55,25 +57,28 @@ public abstract class AbstractTask implements Task {
protected Map<TopicPartition, Long> offsetSnapshotSinceLastFlush = null;
protected final TaskId id;
+ protected final TaskConfig config;
protected final ProcessorTopology topology;
protected final StateDirectory stateDirectory;
protected final ProcessorStateManager stateMgr;
+
private final long taskTimeoutMs;
AbstractTask(final TaskId id,
final ProcessorTopology topology,
final StateDirectory stateDirectory,
final ProcessorStateManager stateMgr,
final Set<TopicPartition> inputPartitions,
- final long taskTimeoutMs,
+ final TaskConfig config,
final String taskType,
final Class<? extends AbstractTask> clazz) {
this.id = id;
this.stateMgr = stateMgr;
this.topology = topology;
+ this.config = config;
this.inputPartitions = inputPartitions;
this.stateDirectory = stateDirectory;
- this.taskTimeoutMs = taskTimeoutMs;
+ this.taskTimeoutMs = config.taskTimeoutMs;
Review Comment:
I would not use a dedicated field for the task timeout but just directly use the task config object `config`.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -80,125 +87,165 @@ void setMainConsumer(final Consumer<byte[], byte[]> mainConsumer) {
this.mainConsumer = mainConsumer;
}
- void handleNewAssignmentAndCreateTasks(final Map<TaskId, Set<TopicPartition>> activeTasksToCreate,
- final Map<TaskId, Set<TopicPartition>> standbyTasksToCreate,
- final Set<TaskId> assignedActiveTasks,
- final Set<TaskId> assignedStandbyTasks) {
- activeTaskCreator.removeRevokedUnknownTasks(assignedActiveTasks);
- standbyTaskCreator.removeRevokedUnknownTasks(assignedStandbyTasks);
- createTasks(activeTasksToCreate, standbyTasksToCreate);
+ void purgePendingTasks(final Set<TaskId> assignedActiveTasks, final Set<TaskId> assignedStandbyTasks) {
+ pendingActiveTasks.keySet().retainAll(assignedActiveTasks);
+ pendingStandbyTasks.keySet().retainAll(assignedStandbyTasks);
+ }
+
+ void addActivePendingTasks(final Map<TaskId, Set<TopicPartition>> pendingTasks) {
+ pendingActiveTasks.putAll(pendingTasks);
+ }
+
+ void addStandbyPendingTasks(final Map<TaskId, Set<TopicPartition>> pendingTasks) {
+ pendingStandbyTasks.putAll(pendingTasks);
}
- void maybeCreateTasksFromNewTopologies(final Set<String> currentNamedTopologies) {
+ void createPendingTasks(final Set<String> currentNamedTopologies) {
createTasks(
- activeTaskCreator.uncreatedTasksForTopologies(currentNamedTopologies),
- standbyTaskCreator.uncreatedTasksForTopologies(currentNamedTopologies)
+ pendingActiveTasksForTopologies(currentNamedTopologies),
+ pendingStandbyTasksForTopologies(currentNamedTopologies)
);
}
- double totalProducerBlockedTime() {
- return activeTaskCreator.totalProducerBlockedTime();
+ private Map<TaskId, Set<TopicPartition>> pendingActiveTasksForTopologies(final Set<String> currentTopologies) {
+ return filterMap(pendingActiveTasks, t -> currentTopologies.contains(t.getKey().topologyName()));
+ }
+
+ private Map<TaskId, Set<TopicPartition>> pendingStandbyTasksForTopologies(final Set<String> currentTopologies) {
+ return filterMap(pendingStandbyTasks, t -> currentTopologies.contains(t.getKey().topologyName()));
}
void createTasks(final Map<TaskId, Set<TopicPartition>> activeTasksToCreate,
final Map<TaskId, Set<TopicPartition>> standbyTasksToCreate) {
+ createActiveTasks(activeTasksToCreate);
+ createStandbyTasks(standbyTasksToCreate);
+ }
+
+ private void createActiveTasks(final Map<TaskId, Set<TopicPartition>> activeTasksToCreate) {
for (final Map.Entry<TaskId, Set<TopicPartition>> taskToBeCreated : activeTasksToCreate.entrySet()) {
final TaskId taskId = taskToBeCreated.getKey();
if (activeTasksPerId.containsKey(taskId)) {
throw new IllegalStateException("Attempted to create an active task that we already own: " + taskId);
}
- }
- for (final Map.Entry<TaskId, Set<TopicPartition>> taskToBeCreated : standbyTasksToCreate.entrySet()) {
- final TaskId taskId = taskToBeCreated.getKey();
-
- if (standbyTasksPerId.containsKey(taskId)) {
- throw new IllegalStateException("Attempted to create a standby task that we already own: " + taskId);
+ if (pendingStandbyTasks.containsKey(taskId)) {
+ throw new IllegalStateException("Attempted to create an active task while we already own its standby: " + taskId);
}
}
- // keep this check to simplify testing (ie, no need to mock `activeTaskCreator`)
if (!activeTasksToCreate.isEmpty()) {
- // TODO: change type to `StreamTask`
for (final Task activeTask : activeTaskCreator.createTasks(mainConsumer, activeTasksToCreate)) {
activeTasksPerId.put(activeTask.id(), activeTask);
- allTasksPerId.put(activeTask.id(), activeTask);
+ pendingActiveTasks.remove(activeTask.id());
for (final TopicPartition topicPartition : activeTask.inputPartitions()) {
activeTasksPerPartition.put(topicPartition, activeTask);
}
}
}
+ }
+
+ private void createStandbyTasks(final Map<TaskId, Set<TopicPartition>> standbyTasksToCreate) {
+ for (final Map.Entry<TaskId, Set<TopicPartition>> taskToBeCreated : standbyTasksToCreate.entrySet()) {
+ final TaskId taskId = taskToBeCreated.getKey();
+
+ if (standbyTasksPerId.containsKey(taskId)) {
+ throw new IllegalStateException("Attempted to create an active task that we already own: " + taskId);
+ }
+
+ if (pendingActiveTasks.containsKey(taskId)) {
+ throw new IllegalStateException("Attempted to create an active task while we already own its standby: " + taskId);
+ }
+ }
- // keep this check to simplify testing (ie, no need to mock `standbyTaskCreator`)
if (!standbyTasksToCreate.isEmpty()) {
- // TODO: change type to `StandbyTask`
for (final Task standbyTask : standbyTaskCreator.createTasks(standbyTasksToCreate)) {
standbyTasksPerId.put(standbyTask.id(), standbyTask);
- allTasksPerId.put(standbyTask.id(), standbyTask);
+ pendingActiveTasks.remove(standbyTask.id());
+ }
+ }
+ }
+
+ void removeTask(final Task taskToRemove) {
+ final TaskId taskId = taskToRemove.id();
+
+ if (taskToRemove.state() != Task.State.CLOSED) {
+ throw new IllegalStateException("Attempted to remove an task that is not closed: " + taskId);
Review Comment:
```suggestion
throw new IllegalStateException("Attempted to remove a task that is not closed: " + taskId);
```
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StandbyTask.java:
##########
@@ -234,6 +236,43 @@ public void closeCleanAndRecycleState() {
log.info("Closed clean and recycled state");
}
+ /**
+ * Create an active task from this standby task without closing and re-initializing the state stores.
+ * The task should have been in suspended state when calling this function
+ *
+ * TODO: we should be able to not need the input partitions as input param in future but always reuse
+ * the task's input partitions when we have fixed partitions -> tasks mapping
+ */
+ public StreamTask recycle(final Time time,
Review Comment:
Could you add unit tests for this method?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1012,14 +1048,16 @@ Set<TaskId> standbyTaskIds() {
.collect(Collectors.toSet());
}
- Map<TaskId, Task> tasks() {
+ Map<TaskId, Task> allTasks() {
// not bothering with an unmodifiable map, since the tasks themselves are mutable, but
// if any outside code modifies the map or the tasks, it would be a severe transgression.
- return tasks.tasksPerId();
+ return tasks.allTasksPerId();
}
Map<TaskId, Task> notPausedTasks() {
Review Comment:
Could you please add unit tests for this method? I think you can reuse the one that was deleted in `TasksTest`.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -573,6 +572,45 @@ public void closeCleanAndRecycleState() {
log.info("Closed clean and recycled state");
}
+ /**
+ * Create a standby task from this active task without closing and re-initializing the state stores.
+ * The task should have been in suspended state when calling this function
+ *
+ * TODO: we should be able to not need the input partitions as input param in future but always reuse
+ * the task's input partitions when we have fixed partitions -> tasks mapping
+ */
+ public StandbyTask recycle(final Set<TopicPartition> inputPartitions) {
Review Comment:
Could you add unit tests for this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r926372626
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -262,51 +294,30 @@ Collection<Task> tasks(final Collection<TaskId> taskIds) {
// TODO: change return type to `StreamTask`
Collection<Task> activeTasks() {
- return readOnlyActiveTasks;
- }
-
- Collection<Task> allTasks() {
- return readOnlyTasks;
- }
-
- Collection<Task> notPausedActiveTasks() {
- return new ArrayList<>(readOnlyActiveTasks)
- .stream()
- .filter(t -> !topologyMetadata.isPaused(t.id().topologyName()))
- .collect(Collectors.toList());
+ return Collections.unmodifiableCollection(activeTasksPerId.values());
}
- Collection<Task> notPausedTasks() {
- return new ArrayList<>(readOnlyTasks)
- .stream()
- .filter(t -> !topologyMetadata.isPaused(t.id().topologyName()))
- .collect(Collectors.toList());
+ /**
+ * All tasks returned by any of the getters are read-only and should NOT be modified;
+ * and the returned task could be modified by other threads concurrently
+ */
+ Set<Task> allTasks() {
+ return union(HashSet::new, new HashSet<>(activeTasksPerId.values()), new HashSet<>(standbyTasksPerId.values()));
}
- Set<TaskId> activeTaskIds() {
- return readOnlyActiveTaskIds;
+ Set<TaskId> allTaskIds() {
+ return union(HashSet::new, activeTasksPerId.keySet(), standbyTasksPerId.keySet());
}
- Set<TaskId> standbyTaskIds() {
- return readOnlyStandbyTaskIds;
- }
-
- // TODO: change return type to `StreamTask`
- Map<TaskId, Task> activeTaskMap() {
- return readOnlyActiveTasksPerId;
- }
-
- // TODO: change return type to `StandbyTask`
- Map<TaskId, Task> standbyTaskMap() {
- return readOnlyStandbyTasksPerId;
- }
-
- Map<TaskId, Task> tasksPerId() {
- return readOnlyTasksPerId;
+ Map<TaskId, Task> allTasksPerId() {
+ final Map<TaskId, Task> ret = new HashMap<>();
+ ret.putAll(activeTasksPerId);
+ ret.putAll(standbyTasksPerId);
+ return ret;
Review Comment:
I would even make it unmodifiable to make it clear that changing the returned map will not change the maps within the `Tasks` object.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -262,51 +294,30 @@ Collection<Task> tasks(final Collection<TaskId> taskIds) {
// TODO: change return type to `StreamTask`
Collection<Task> activeTasks() {
- return readOnlyActiveTasks;
- }
-
- Collection<Task> allTasks() {
- return readOnlyTasks;
- }
-
- Collection<Task> notPausedActiveTasks() {
- return new ArrayList<>(readOnlyActiveTasks)
- .stream()
- .filter(t -> !topologyMetadata.isPaused(t.id().topologyName()))
- .collect(Collectors.toList());
+ return Collections.unmodifiableCollection(activeTasksPerId.values());
}
- Collection<Task> notPausedTasks() {
- return new ArrayList<>(readOnlyTasks)
- .stream()
- .filter(t -> !topologyMetadata.isPaused(t.id().topologyName()))
- .collect(Collectors.toList());
+ /**
+ * All tasks returned by any of the getters are read-only and should NOT be modified;
+ * and the returned task could be modified by other threads concurrently
+ */
+ Set<Task> allTasks() {
+ return union(HashSet::new, new HashSet<>(activeTasksPerId.values()), new HashSet<>(standbyTasksPerId.values()));
}
- Set<TaskId> activeTaskIds() {
- return readOnlyActiveTaskIds;
+ Set<TaskId> allTaskIds() {
+ return union(HashSet::new, activeTasksPerId.keySet(), standbyTasksPerId.keySet());
}
- Set<TaskId> standbyTaskIds() {
- return readOnlyStandbyTaskIds;
- }
-
- // TODO: change return type to `StreamTask`
- Map<TaskId, Task> activeTaskMap() {
- return readOnlyActiveTasksPerId;
- }
-
- // TODO: change return type to `StandbyTask`
- Map<TaskId, Task> standbyTaskMap() {
- return readOnlyStandbyTasksPerId;
- }
-
- Map<TaskId, Task> tasksPerId() {
- return readOnlyTasksPerId;
+ Map<TaskId, Task> allTasksPerId() {
+ final Map<TaskId, Task> ret = new HashMap<>();
Review Comment:
nit: `ret` is not really a meaningful variable name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r925882848
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -34,29 +33,40 @@
import java.util.TreeMap;
import java.util.stream.Collectors;
+import static org.apache.kafka.common.utils.Utils.filterMap;
+import static org.apache.kafka.common.utils.Utils.union;
+
+/**
+ * All tasks contained by the Streams instance.
+ *
+ * Note that these tasks are shared between the TaskManager (stream thread) and the StateUpdater (restore thread),
+ * i.e. all running active tasks are processed by the former and all restoring active tasks and standby tasks are
+ * processed by the latter.
+ */
class Tasks {
private final Logger log;
- private final TopologyMetadata topologyMetadata;
-
- private final Map<TaskId, Task> allTasksPerId = Collections.synchronizedSortedMap(new TreeMap<>());
- private final Map<TaskId, Task> readOnlyTasksPerId = Collections.unmodifiableMap(allTasksPerId);
- private final Collection<Task> readOnlyTasks = Collections.unmodifiableCollection(allTasksPerId.values());
// TODO: change type to `StreamTask`
private final Map<TaskId, Task> activeTasksPerId = new TreeMap<>();
+ // TODO: change type to `StandbyTask`
+ private final Map<TaskId, Task> standbyTasksPerId = new TreeMap<>();
+
+ // Tasks may have been assigned for a NamedTopology that is not yet known by this host. When that occurs we stash
+ // these unknown tasks until either the corresponding NamedTopology is added and we can create them at last, or
+ // we receive a new assignment and they are revoked from the thread.
+
+ // Tasks may have been assigned but not yet created because:
+ // 1. They are for a NamedTopology that is yet known by this host.
+ // 2. They are to be recycled from an existing restoring task yet to be returned from the task updater.
Review Comment:
Oops, my bad :) thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12397: KAFKA-10199: Cleanup TaskManager and Task interfaces
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12397:
URL: https://github.com/apache/kafka/pull/12397#discussion_r925945110
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/PartitionGroup.java:
##########
@@ -216,8 +216,9 @@ long partitionTimestamp(final TopicPartition partition) {
}
// creates queues for new partitions, removes old queues, saves cached records for previously assigned partitions
- void updatePartitions(final Set<TopicPartition> newInputPartitions, final Function<TopicPartition, RecordQueue> recordQueueCreator) {
+ void updatePartitions(final Set<TopicPartition> inputPartitions, final Function<TopicPartition, RecordQueue> recordQueueCreator) {
Review Comment:
@cadonna This is a bug exposed from the unit test now: we should not modify the passed in params directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org