You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "xuzhiwen1255 (via GitHub)" <gi...@apache.org> on 2023/02/24 07:46:54 UTC
[GitHub] [flink] xuzhiwen1255 opened a new pull request, #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
xuzhiwen1255 opened a new pull request, #22010:
URL: https://github.com/apache/flink/pull/22010
## What is the purpose of the change
In the sequence mode, the dataGen field will preload all the sequence values when it is initialized, and it will take a long time during this period
## Brief change log
- Adapt the way of preloading the value to generate a sequence value every time the next method is called
- The checkpoint saves not all the sequence values, but the position where the current task sends the sequence value
## Verifying this change
`DataGeneratorSourceTest` and `DataGenTableSourceFactoryTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1484436957
@reswqa CI is fine now, can you come over and have a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1206745426
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
Thanks for sharing your view, @RyanSkraba. I think you're right - the implementation in master (i.e. without this PR's change) doesn't support reducing the parallelism properly, too.
> Avoid performance skew after reducing parallelism. Currently, going from N tasks to N-1 tasks will double the length of time it takes, since one task will have twice as many values to emit as the others. This is a undesirable consequence of this PR, but can probably also be discussed and implemented in a subsequent step.
I don't understand that part: Isn't the performance skew issue you're describing more general and not rely connected to the `DataGenerator`? The skew you're mentioning is happening in any case when reducing the parallelism. It's more of a question how Flink would distribute the operator state after the parallelism was reduced. :thinking: Therefore, I don't see this as a problem for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1552339085
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] RyanSkraba commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1200460171
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,40 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence =
Review Comment:
```suggestion
long nextSequence =
```
A really minor suggestion, but I'd inverse these variable names for readability!
The one stored in the InternalState is the *next* or future value (to be emitted on a later call): maybe `futureValue`?
The one currently being store in this local variable is the `currentValue`.
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,40 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence =
+ state.currentValue == 0 ? this.start + state.taskId : state.currentValue;
+ state.currentValue = nextSequence + state.stepSize;
+ // All sequence values are cleared from the stateList after they have been sent
+ if (state.currentValue > this.end) {
Review Comment:
If this.end is close to `Long.MAX_VALUE`, there's a possibility of overflow here with `nextSequence + state.stepSize`.
```suggestion
if (state.currentValue > this.end || state.currentValue <= nextSequence) {
```
(I think this is correct logic, but `Math.addExact` could be used as well.)
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,40 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence =
+ state.currentValue == 0 ? this.start + state.taskId : state.currentValue;
Review Comment:
More seriously, there's a problem with using `state.currentValue == 0` as the unitialized state. Imagine if you're dumping the range from -1000L to 1000L, when the task finds itself *meant* to be emitting zero, it will start over from it's first value instead.
I'm working through it a bit, but it might be enough just to set it in the InternalState from the start: `new InternalState(taskIdx, stepSize, start + taskIdx)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1564371012
> I think about it, the change of parallelism already exists in the existing unit test, I think we don't need to add the test, because it already includes the complete generated test case of the sequence.
Thanks for pointing that out. I missed `DataGeneratorSourceTest`. Do you still see the value in having a `SequenceGeneratorTest` testing the basic functionality of the `SequenceGenerator`, though?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1566386306
> > I think about it, the change of parallelism already exists in the existing unit test, I think we don't need to add the test, because it already includes the complete generated test case of the sequence.
>
> Thanks for pointing that out. I missed `DataGeneratorSourceTest`. Do you still see the value in having a `SequenceGeneratorTest` testing the basic functionality of the `SequenceGenerator`, though? It might help to understand the contract of the `SequenceGenerator` class.
I don't think it is necessary to use `SequenceGeneratorTest` for testing, because the existing tests have covered its basic functions. Later, we may need to test how to avoid data skew and test sequence generation integrity when adjusting operator parallelism.
@XComp WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1261890872
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -22,29 +22,44 @@
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
-import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.runtime.state.FunctionSnapshotContext;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.shaded.guava30.com.google.common.collect.Queues;
+
import java.math.BigDecimal;
import java.math.MathContext;
import java.math.RoundingMode;
-import java.util.ArrayDeque;
-import java.util.Deque;
+import java.util.ArrayList;
+import java.util.NoSuchElementException;
+import java.util.Queue;
/**
* A stateful, re-scalable {@link DataGenerator} that emits each number from a given interval
* exactly once, possibly in parallel.
+ *
+ * <p>It maintains a state internally to record the position of the current subtask sending
+ * sequence. When the task resumes, it will continue to send the sequence value according to the
+ * position sent by the state, until all the sequences have been sent.
+ *
+ * <p><b>IMPORTANT NOTE: </b> When the degree of parallelism increases, there may be cases where
+ * subtasks are running empty. When the degree of parallelism decreases, there may be cases where
+ * one subtask handles multiple states.
Review Comment:
And we use the priority queue to store state, so each subtask can be guaranteed to be sent according to the size of the sequence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1192906502
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +72,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ totalNoOfElements = Math.abs(end - start + 1);
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ internalStates.add(new InternalState(0, taskIdx, stepSize));
+ }
+ }
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ private long toCollect(long baseSize, long stepSize, int taskIdx) {
+ return (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence = state.collected * state.stepSize + (start + state.taskId);
+ state.collected++;
Review Comment:
Let me rethink, perhaps it is more reasonable to change this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1201668515
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,40 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence =
+ state.currentValue == 0 ? this.start + state.taskId : state.currentValue;
Review Comment:
Thanks for the reminder, I didn't think it was possible to use a negative number as a starting value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1463322542
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1608853554
> Hi @xuzhiwen1255 , sorry for making you wait for so long. I was on vacation and have to focus on some other work right now. I try to get back to this PR soon'ish
@XComp It's all right, have a nice holiday.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1192906438
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +72,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ totalNoOfElements = Math.abs(end - start + 1);
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ internalStates.add(new InternalState(0, taskIdx, stepSize));
+ }
+ }
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ private long toCollect(long baseSize, long stepSize, int taskIdx) {
+ return (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence = state.collected * state.stepSize + (start + state.taskId);
+ state.collected++;
+ // All sequence values are cleared from the stateList after they have been sent
+ if (state.collected
+ >= toCollect(
+ safeDivide(totalNoOfElements, state.stepSize),
+ state.stepSize,
+ state.taskId)) {
+ iterator.remove();
}
+ return nextSequence;
}
+
+ // Before calling this method, you should call hasNext to check
+ throw new IllegalStateException();
Review Comment:
You're quite right. I'll change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1205131529
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
Review Comment:
I think that's a better name for it, yes. :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1561500331
On another note: We should rebase the branch to include FLINK-31962 which would make CI succeed again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1463436077
@reswqa Re-running the cli is no problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1261873363
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -22,29 +22,44 @@
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
-import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.runtime.state.FunctionSnapshotContext;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.shaded.guava30.com.google.common.collect.Queues;
+
import java.math.BigDecimal;
import java.math.MathContext;
import java.math.RoundingMode;
-import java.util.ArrayDeque;
-import java.util.Deque;
+import java.util.ArrayList;
+import java.util.NoSuchElementException;
+import java.util.Queue;
/**
* A stateful, re-scalable {@link DataGenerator} that emits each number from a given interval
* exactly once, possibly in parallel.
+ *
+ * <p>It maintains a state internally to record the position of the current subtask sending
+ * sequence. When the task resumes, it will continue to send the sequence value according to the
+ * position sent by the state, until all the sequences have been sent.
+ *
+ * <p><b>IMPORTANT NOTE: </b> When the degree of parallelism increases, there may be cases where
+ * subtasks are running empty. When the degree of parallelism decreases, there may be cases where
+ * one subtask handles multiple states.
Review Comment:
I don't think so, because the current state is also out of order, because we can't guarantee the same processing speed of all subtasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi… [flink]
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1959219838
I think so. I will modify it as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi… [flink]
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1869194665
@XComp I'm sorry that I haven't had time to deal with this problem recently due to some things. Now I plan to continue to deal with this issue. Do you have time to take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1201664511
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,40 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence =
Review Comment:
How about calling it nextValue, I think it's more reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] RyanSkraba commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1192511163
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +72,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ totalNoOfElements = Math.abs(end - start + 1);
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ internalStates.add(new InternalState(0, taskIdx, stepSize));
+ }
+ }
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ private long toCollect(long baseSize, long stepSize, int taskIdx) {
+ return (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence = state.collected * state.stepSize + (start + state.taskId);
+ state.collected++;
+ // All sequence values are cleared from the stateList after they have been sent
+ if (state.collected
+ >= toCollect(
+ safeDivide(totalNoOfElements, state.stepSize),
+ state.stepSize,
+ state.taskId)) {
+ iterator.remove();
}
+ return nextSequence;
}
+
+ // Before calling this method, you should call hasNext to check
+ throw new IllegalStateException();
Review Comment:
```suggestion
throw new IllegalStateException("SequenceGenerator.nextValue() was called with no remaining values.");
```
This code should be unreachable, so it's not a big deal -- but just in case, a helpful message is a good practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1257913768
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -189,8 +206,28 @@ public static SequenceGenerator<String> stringGenerator(long start, long end) {
return new SequenceGenerator<String>(start, end) {
@Override
public String next() {
- return valuesToEmit.poll().toString();
+ return nextValue().toString();
}
};
}
+
+ /**
+ * The internal state of the sequence generator, which is used to record the latest state of the
+ * sequence value sent by the current sequence generator. When recovering from the state, it is
+ * guaranteed to continue sending the sequence value from the latest state.
+ */
+ private static class InternalState implements Comparable<InternalState> {
Review Comment:
What's your opinion on renaming this internal class into `SubTaskState`? `InternalState` sounds too generic. :thinking:
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +80,45 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
Review Comment:
```suggestion
final ListStateDescriptor<InternalState> stateDescriptor =
```
There are a few other code locations where we could make use of the `final` keyword for local variables.
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -22,29 +22,44 @@
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
-import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.runtime.state.FunctionSnapshotContext;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.shaded.guava30.com.google.common.collect.Queues;
+
import java.math.BigDecimal;
import java.math.MathContext;
import java.math.RoundingMode;
-import java.util.ArrayDeque;
-import java.util.Deque;
+import java.util.ArrayList;
+import java.util.NoSuchElementException;
+import java.util.Queue;
/**
* A stateful, re-scalable {@link DataGenerator} that emits each number from a given interval
* exactly once, possibly in parallel.
+ *
+ * <p>It maintains a state internally to record the position of the current subtask sending
+ * sequence. When the task resumes, it will continue to send the sequence value according to the
+ * position sent by the state, until all the sequences have been sent.
+ *
+ * <p><b>IMPORTANT NOTE: </b> When the degree of parallelism increases, there may be cases where
+ * subtasks are running empty. When the degree of parallelism decreases, there may be cases where
+ * one subtask handles multiple states.
Review Comment:
Shall we also add that decreasing the parallelism might result in an out-of-orderness of the sequence?
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +80,45 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Queues.newPriorityQueue();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.offer(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
- final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
-
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ // The first time the job is executed.
+ final int startOffset = runtimeContext.getIndexOfThisSubtask();
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(stepSize, start + startOffset);
+ internalStates.offer(state);
+ }
+ }
+
+ public Long nextValue() {
+ if (internalStates.isEmpty()) {
+ // Before calling nextValue method, you should call hasNext to check.
Review Comment:
```suggestion
```
This comment doesn't add any value, don't you think? :thinking: If you want to give this hint to the developer, you might want to add it to the error message. Otherwise, you're expecting the developer to do a code investigation to find the hint of this comment.
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -22,29 +22,44 @@
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
-import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.runtime.state.FunctionSnapshotContext;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.shaded.guava30.com.google.common.collect.Queues;
Review Comment:
```suggestion
import org.apache.flink.shaded.guava31.com.google.common.collect.Queues;
```
This won't work anymore because the `flink-shaded` dependencies were upgraded (FLINK-32032). Rebasing the branch might reveal other affected code locations.
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +80,45 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Queues.newPriorityQueue();
Review Comment:
```suggestion
this.internalStates = Queues.newPriorityQueue();
```
Can't we make `internalStates` a `final` field? We don't need to re-instantiate it. I'm then wondering whether we could add a Precondition that checks here that the queue is empty when calling the `open` method or whether there are situations where the queue could still have entries (in which case we would have to call `clear()`)?
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -22,29 +22,44 @@
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
-import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.runtime.state.FunctionSnapshotContext;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.shaded.guava30.com.google.common.collect.Queues;
+
import java.math.BigDecimal;
import java.math.MathContext;
import java.math.RoundingMode;
-import java.util.ArrayDeque;
-import java.util.Deque;
+import java.util.ArrayList;
+import java.util.NoSuchElementException;
+import java.util.Queue;
/**
* A stateful, re-scalable {@link DataGenerator} that emits each number from a given interval
* exactly once, possibly in parallel.
+ *
+ * <p>It maintains a state internally to record the position of the current subtask sending
+ * sequence. When the task resumes, it will continue to send the sequence value according to the
+ * position sent by the state, until all the sequences have been sent.
+ *
+ * <p><b>IMPORTANT NOTE: </b> When the degree of parallelism increases, there may be cases where
+ * subtasks are running empty. When the degree of parallelism decreases, there may be cases where
+ * one subtask handles multiple states.
*/
@Experimental
public abstract class SequenceGenerator<T> implements DataGenerator<T> {
private final long start;
private final long end;
+ /**
+ * Save the intermediate state of the data to be sent by the current subtask,when the state
Review Comment:
```suggestion
* Save the intermediate state of the data to be sent by the current subtask, when the state
```
nit
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -189,8 +206,28 @@ public static SequenceGenerator<String> stringGenerator(long start, long end) {
return new SequenceGenerator<String>(start, end) {
@Override
public String next() {
- return valuesToEmit.poll().toString();
+ return nextValue().toString();
}
};
}
+
+ /**
+ * The internal state of the sequence generator, which is used to record the latest state of the
Review Comment:
```suggestion
* The internal state of the sequence generator's subtask(s), which is used to record the latest state of the
```
Is this more specific?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1261875654
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -189,8 +206,28 @@ public static SequenceGenerator<String> stringGenerator(long start, long end) {
return new SequenceGenerator<String>(start, end) {
@Override
public String next() {
- return valuesToEmit.poll().toString();
+ return nextValue().toString();
}
};
}
+
+ /**
+ * The internal state of the sequence generator, which is used to record the latest state of the
+ * sequence value sent by the current sequence generator. When recovering from the state, it is
+ * guaranteed to continue sending the sequence value from the latest state.
+ */
+ private static class InternalState implements Comparable<InternalState> {
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1442997250
@reswqa Can you review it for me, Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi… [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1497437961
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -66,33 +91,47 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ final ListStateDescriptor<SubTaskState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(SubTaskState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (!subTaskStates.isEmpty()) {
+ this.subTaskStates.clear();
+ }
Review Comment:
```suggestion
this.subTaskStates.clear();
```
nit: clearing an empty collection shouldn't hurt.
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -101,26 +140,20 @@ public void snapshotState(FunctionSnapshotContext context) throws Exception {
this.checkpointedState != null,
"The " + getClass().getSimpleName() + " state has not been properly initialized.");
- this.checkpointedState.update(new ArrayList<>(this.valuesToEmit));
+ this.checkpointedState.clear();
+ this.checkpointedState.addAll(new ArrayList<>(subTaskStates));
Review Comment:
```suggestion
this.checkpointedState.update(new ArrayList<>(subTaskStates));
```
nit: we can continue to rely on `update` here, I guess, to keep the diff smaller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi… [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-2771459861
This PR has been closed since it has not had any activity in 120 days.
If you feel like this was a mistake, or you would like to continue working on it,
please feel free to re-open the PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1208764798
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
> That is to say, instead of having N internal states when there are N tasks initially, we would have 2N, or 10N, or 100N. That number doesn't have to be extremely high to avoid skew here, and still benefit from the "concise" and efficient InternalState you've implemented here.
Yes, but I haven't figured out how to solve this problem reasonably.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1702228065
No worries. Especially considering that I won't be that responsive in September in general. I just wanted to let you know to make prioritization of work easier for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1563739704
> In addition to the comments, I feel like we're lacking some tests here. Is there a way to unit test the generator (maybe even in the context of a parallelism change)?
I think about it, the change of parallelism already exists in the existing unit test, I think we don't need to add the test, because it already includes the complete generated test case of the sequence.
> In this DataGeneratorSourceTest#innerTestDataGenCheckpointRestore test method, the degree of parallelism is modified .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1206414704
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
> * Avoid performance skew after reducing parallelism. Currently, going from N tasks to N-1 tasks will double the length of time it takes, since one task will have twice as many values to emit as the others. This is a undesirable consequence of this PR, but can probably also be discussed and implemented in a subsequent step.
Yes, I noticed this problem, but I haven't thought of a better solution.
But I have an idea, that is, for each subtask, we let him generate more InternalState, and the general process is described as follows:
Assuming that the parallelism is 5 and sending 100 pieces of data, we let each subtaks be logically divided into multiple InternalStates
The current InternalState is as follows
```java
InternalState{
long stepSize;
long nextValue;
}
```
When subtask-0 is initialized: only one `InternalState(5,0)` will be initialized;
So my idea is to add an end to indicate the end value of the current InternalState
```java
InternalState{
long stepSize;
long nextValue;
long end;
}
```
Cutting when subtask-0 is initialized produces two: `InternalState(5,0,50)`, `InternalState(5,0,100)`;
Of course, it doesn’t have to be two, it can be multiple, so that when the parallelism is readjusted, one task can avoid processing multiple InternalStates.
@XComp @RyanSkraba WDYT?
At that time, I will raise a jira to track this issue, and then we will discuss this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1637414390
> I'm not a big fan of adding tests that belong in a PR into a separate PR. You could just create the test class in this PR with the set of tests that fit to this PR. PR #21971 should then add additional tests related to its changes. The conflict resolution shouldn't be too complex because your adding independent test methods.
OK let me add a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1443013003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4a9ee58a62b1b2e4d540ea6293554638722e53",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f4a9ee58a62b1b2e4d540ea6293554638722e53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f4a9ee58a62b1b2e4d540ea6293554638722e53 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1553949885
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1204910922
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
Yes, multiple states will only appear when the degree of parallelism is reduced.
I guess what you're thinking is: are we going to get the latest sequence value by putting the state into the queue and then taking it off the head of the queue each time and putting it at the end? Instead of sending all the sequences of a state and then sending the sequence value of the next state?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1608851928
Hi @xuzhiwen1255 , sorry for making you wait for so long. I was on vacation and have to focus on some other work right now. I try to get back to this PR soon'ish
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1566388564
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi… [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-2593617520
This PR is being marked as stale since it has not had any activity in the last 180 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
<p>
If you are having difficulty finding a reviewer, please reach out to the
[community](https://flink.apache.org/what-is-flink/community/).
<p>
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 90 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xzw0223 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xzw0223 (via GitHub)" <gi...@apache.org>.
xzw0223 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1608808651
@XComp @RyanSkraba Any more questions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1205133307
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
Yes, it feels like `internalState` should be some kind of priority queue that serves the internal state with the lowest element. I really would prefer having this put into some kind of unit test as well. :thinking: WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] RyanSkraba commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1206364163
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
This seems like a good idea, and it's probably appropriate to create some JIRAs to implement in subsequent, incremental steps.
The current internal states (before this PR) for a generator from 1 to 1M would have 1 million values: `1,2,3...1000000`. If it were checkpointed, it would **(1)** take quite a bit of space and **(2)** I'm not sure that restoring it to a lower parallelism retains any lowest-first semantics.
This PR solves **(1)**, but doesn't change much in terms of **(2)**. The internal states would be `(1,3), (2,3), (3,3)` (assuming a parallelism of 3).
If it was restored to a parallelism of two (after the PR), and one task got `(1,3), (2,3)` as its internal states, it would emit `1,4,7,10 ... 1000000, 2, 5, 8 ...`.
If my understanding is correct, emitting `2` _after_ `1000000` is a possibility in both scenarios. There's an opportunity here:
* Improve the logic so that a task always emits the lowest of the possible values from the internal states that it possesses (although a priority queue is probably unnecessary, it should end up roughly equivalent to sorting and cycling through the states). This is really a new improvement that could be applied to the current state.
* Avoid performance skew after reducing parallelism. Currently, going from N tasks to N-1 tasks will double the length of time it takes, since one task will have twice as many values to emit as the others. This is a undesirable consequence of this PR, but can probably also be discussed and implemented in a subsequent step.
One thing that I really like about this PR is that it enables huge sequences that aren't possible today, in an efficient manner. This is a really compelling advantage over the current state in every way!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1206414704
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
> * Avoid performance skew after reducing parallelism. Currently, going from N tasks to N-1 tasks will double the length of time it takes, since one task will have twice as many values to emit as the others. This is a undesirable consequence of this PR, but can probably also be discussed and implemented in a subsequent step.
Yes, I noticed this problem, but I haven't thought of a better solution.
But I have an idea, that is, for each subtask, we let him generate more InternalState, and the general process is described as follows:
Assuming that the parallelism is 5 and sending 100 pieces of data, we let each subtaks be logically divided into multiple InternalStates
The current InternalState is as follows
```java
InternalState{
long stepSize;
long nextValue;
}
```
When subtask-0 is initialized: only one `InternalState(5,0)` will be initialized;
So my idea is to add an end to indicate the end value of the current InternalState
```java
InternalState{
long stepSize;
long nextValue;
long end;
}
```
Cutting when subtask-0 is initialized produces two: `InternalState(5,0,50)`, `InternalState(5,0,100)`;
Of course, it doesn’t have to be two, it can be multiple, so that when the parallelism is readjusted, one task can avoid processing multiple InternalStates.
@XComp @RyanSkraba WDYT?
At that time, I will raise a jira to track this issue, and then we will discuss this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1206414704
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
> * Avoid performance skew after reducing parallelism. Currently, going from N tasks to N-1 tasks will double the length of time it takes, since one task will have twice as many values to emit as the others. This is a undesirable consequence of this PR, but can probably also be discussed and implemented in a subsequent step.
Yes, I noticed this problem, but I haven't thought of a better solution.
But I have an idea, that is, for each subtask, we let him generate more InternalState, and the general process is described as follows:
Assuming that the parallelism is 5 and sending 100 pieces of data, we let each subtaks be logically divided into multiple InternalStates
The current InternalState is as follows
```java
InternalState{
long stepSize;
long nextValue;
}
```
When subtask-0 is initialized: only one InternalState(5,0) will be initialized;
So my idea is to add an end to indicate the end value of the current InternalState
```java
InternalState{
long stepSize;
long nextValue;
long end;
}
```
Cutting when subtask-0 is initialized produces two: InternalState(5,0,50), InternalState(5,0,100);
Of course, it doesn’t have to be two, it can be multiple, so that when the parallelism is readjusted, one task can avoid processing multiple InternalStates.
@XComp @RyanSkraba WDYT?
At that time, I will raise a jira to track this issue, and then we will discuss this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1701325710
> Any updates on that one @xuzhiwen1255 ? 🙂
@XComp Sorry, the company has been busy iterating recently. I will update it in the next few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1204923081
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
Review Comment:
Its function is to distinguish the starting value of each degree of parallelism according to start.
I think it could be called `startOffset`, the offset used to determine the start value of each task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1556116616
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] RyanSkraba commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1191359995
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +72,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ totalNoOfElements = Math.abs(end - start + 1);
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ internalStates.add(new InternalState(0, taskIdx, stepSize));
+ }
+ }
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ private long toCollect(long baseSize, long stepSize, int taskIdx) {
+ return (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence = state.collected * state.stepSize + (start + state.taskId);
+ state.collected++;
Review Comment:
If we are updating the current state at every iteration anyway: instead of incrementing `collected`, why not just retain the value in the current state? It's easy to calculate the *next* (just add stepSize) and you know to drop the state from the list if it goes over `this.end`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] RyanSkraba commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1206825892
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
@xuzhiwen1255 This sounds perfect!
> But I have an idea, that is, for each subtask, we let him generate more InternalState
That is to say, instead of having N internal states when there are N tasks initially, we would have 2N, or 10N, or 100N. That number doesn't have to be extremely high to avoid skew here, and still benefit from the "concise" and efficient InternalState you've implemented here.
I think there might be some cool way of splitting up the ranges differently, but I'd like to think about it! This should probably go on the future JIRA though :D
@XComp Yes, this issue would happen to any source that created "one split per task" on initialization, where the split corresponds to the operator state -- but the previous implementation didn't do that. It was one state per element to emit, so it _could_ be perfectly rebalanced (but overall pretty inefficiently).
I agree that it's not a problem for this PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1206745426
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
Thanks for sharing your view, @RyanSkraba. I think you're right - the implementation in master (i.e. without this PR's change) doesn't support reducing the parallelism properly, too.
```
Avoid performance skew after reducing parallelism. Currently, going from N tasks to N-1 tasks will double the length of time it takes, since one task will have twice as many values to emit as the others. This is a undesirable consequence of this PR, but can probably also be discussed and implemented in a subsequent step.
```
I don't understand that part: Isn't the performance skew issue you're describing more general and not rely connected to the `DataGenerator`? The skew you're mentioning is happening in any case when reducing the parallelism. It's more of a question how Flink would distribute the operator state after the parallelism was reduced. :thinking: Therefore, I don't see this as a problem for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1634336399
> In https://github.com/apache/flink/pull/21971 we added SequenceGeneratorTest, I think we can add the corresponding test method in https://github.com/apache/flink/pull/21971
I'm not a big fan of adding tests that belong in a PR into a separate PR. You could just create the test class in this PR with the set of tests that fit to this PR. PR #21971 should then add additional tests related to its changes. The conflict resolution shouldn't be too complex because your adding independent test methods. :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1204355579
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
Review Comment:
It's not that easy to grasp why we use `taskId` in this context. Could we come up with a variable name that describes the purpose of it in a better way (tbh, I'm struggling to come up with a proposal here).
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
Review Comment:
Why do we pass the `taskIdx` as a dedicated parameter? It doesn't look like we're using it anywhere? :thinking:
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long currentValue = state.nextValue;
+ try {
+ state.nextValue = Math.addExact(currentValue, state.stepSize);
+ // All sequence values are cleared from the stateList after they have been sent.
+ if (state.nextValue > this.end) {
+ iterator.remove();
+ }
+ } catch (ArithmeticException e) {
+ // When the limit is exceeded, it means that the data has been sent and needs to
+ // be cleared from the state.
+ iterator.remove();
}
+
+ return currentValue;
}
+
+ // Before calling nextValue method, you should call hasNext to check.
+ throw new IllegalStateException(
Review Comment:
The `nextValue` method is called by `Iterator.next()`. The `Iterator`'s `next()` method has a clear contract for this kind of situation which states that a `NoSuchElementException` should be thrown. We might want to follow that, if I'm not mistaken.
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
When do we have the case that there is more than one state in the `internalStates`? Is that when we're reducing the parallelism and multiple states get picked up by a single operator? Is it enough to always select the first `InternalState`? It feels to me that we're processing one state first until we reach the end and would then process the next state which has a way lower number again? :thinking:
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -42,9 +44,13 @@
private final long start;
private final long end;
+ /**
+ * Save the intermediate state of the data to be sent by the current subtask,when the state
+ * restore, the sequence values continue to be sent based on the intermediate state.
+ */
+ private transient ArrayList<InternalState> internalStates;
Review Comment:
```suggestion
private transient Queue<InternalState> internalStates;
```
nit: Wouldn't `Queue` be the better interface? :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1204467390
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
Review Comment:
I'm also wondering whether we could add some JavaDoc (either to `SequenceGenerator` or to `InternalState`) to describe the logic of this class a bit more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi… [flink]
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1961151386
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi… [flink]
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1962898978
Hi @XComp I have solved the problem, please check again, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] RyanSkraba commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1191321556
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -42,9 +44,14 @@
private final long start;
private final long end;
+ private long totalNoOfElements;
+ /**
+ * Save the intermediate state of the data to be sent by the current subtask,when the state
+ * restore, the sequence values continue to be sent based on the intermediate state.
+ */
+ private ArrayList<InternalState> internalStates;
Review Comment:
Does this need to be transient as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] RyanSkraba commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "RyanSkraba (via GitHub)" <gi...@apache.org>.
RyanSkraba commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1200461236
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,40 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
// the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
+ if (iterator.hasNext()) {
+ InternalState state = iterator.next();
+ long nextSequence =
+ state.currentValue == 0 ? this.start + state.taskId : state.currentValue;
Review Comment:
More seriously, there's a problem with using `state.currentValue == 0` as the unitialized state. Imagine if you're dumping the range from -1000L to 1000L: when the task finds itself *meant* to be emitting zero, it will start over from its first value instead.
I'm working through it a bit, but it might be enough just to set it in the InternalState from the start: `new InternalState(taskIdx, stepSize, start + taskIdx)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1552346563
@RyanSkraba @XComp Please check again, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1205192672
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
+ internalStates.add(state);
+ }
+ }
- for (long collected = 0; collected < toCollect; collected++) {
- this.valuesToEmit.add(collected * stepSize + congruence);
+ public Long nextValue() {
+ Iterator<InternalState> iterator = internalStates.iterator();
Review Comment:
I'll try to put it in unit tests... ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1563768151
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1701320568
Any updates on that one @xuzhiwen1255 :slightly_smiling_face:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1633453018
> I still think that it might make sense to introduce a `SequenceGeneratorTest`. This would allow us to test the edge cases easily (e.g. reaching `Long.MAX_VALUE`) more specifically.
In #21971 we added SequenceGeneratorTest, I think we can add the corresponding test method in #21971
@XComp WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "xuzhiwen1255 (via GitHub)" <gi...@apache.org>.
xuzhiwen1255 commented on code in PR #22010:
URL: https://github.com/apache/flink/pull/22010#discussion_r1204911414
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java:
##########
@@ -65,33 +71,46 @@ public void open(
this.checkpointedState == null,
"The " + getClass().getSimpleName() + " has already been initialized.");
- this.checkpointedState =
- context.getOperatorStateStore()
- .getListState(
- new ListStateDescriptor<>(
- name + "-sequence-state", LongSerializer.INSTANCE));
- this.valuesToEmit = new ArrayDeque<>();
- if (context.isRestored()) {
- // upon restoring
+ ListStateDescriptor<InternalState> stateDescriptor =
+ new ListStateDescriptor<>(
+ name + "-sequence-state", TypeInformation.of(InternalState.class));
+ this.checkpointedState = context.getOperatorStateStore().getListState(stateDescriptor);
+ this.internalStates = Lists.newArrayList();
- for (Long v : this.checkpointedState.get()) {
- this.valuesToEmit.add(v);
- }
+ if (context.isRestored()) {
+ checkpointedState.get().forEach(state -> internalStates.add(state));
} else {
- // the first time the job is executed
- final int stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ // The first time the job is executed.
final int taskIdx = runtimeContext.getIndexOfThisSubtask();
- final long congruence = start + taskIdx;
-
- long totalNoOfElements = Math.abs(end - start + 1);
- final int baseSize = safeDivide(totalNoOfElements, stepSize);
- final int toCollect =
- (totalNoOfElements % stepSize > taskIdx) ? baseSize + 1 : baseSize;
+ final long stepSize = runtimeContext.getNumberOfParallelSubtasks();
+ InternalState state = new InternalState(taskIdx, stepSize, start + taskIdx);
Review Comment:
> Why do we pass the `taskIdx` as a dedicated parameter? It doesn't look like we're using it anywhere? 🤔
Yes it is not used and i will delete it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] reswqa commented on pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on PR #22010:
URL: https://github.com/apache/flink/pull/22010#issuecomment-1446551387
Thanks @xuzhiwen1255 for creating this, It seems that the CI is failure, PTAL first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi… [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #22010: [FLINK-31192][connectors/dataGen] Fix dataGen takes too long to initi…
URL: https://github.com/apache/flink/pull/22010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org