You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/11/09 00:08:23 UTC

[GitHub] [flink] StephanEwen opened a new pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

StephanEwen opened a new pull request #13988:
URL: https://github.com/apache/flink/pull/13988


   ## Purpose / Brief change log
   
   This adds convenience methods for the common actions around split requesting, rather than requiring every source to implement the event handling themselves.
   
     1. To simplify split requests, this adds `SplitReaderContext.sendSplitRequest()` and `SplitEnumerator.handleSplitRequest()`.
     2. To simplify dealing with a finite amount of bounded splits (all batch use cases), this add `SplitEnumeratorContext.signalNoMoreSplits()` and `SourceReader.notifyNoMoreSplits()`.
     3. The PR removes the `RequestSplitEvent` and `NoMoreSplitsEvent`.
   
   ## Verifying this change
   
   This change is API rework and covered by existing tests.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): **no**
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **yes**
     - The serializers: **no**
     - The runtime per-record code paths (performance sensitive): **no**
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **no**
     - The S3 file system connector: **no**
   
   ## Documentation
   
     - Does this pull request introduce a new feature? **no** (only API extensions)
     - If yes, how is the feature documented? **JavaDocs**
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13988:
URL: https://github.com/apache/flink/pull/13988#issuecomment-723691966


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1d621a3f57b64dbbd9f645347267aef8b245ca74",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9324",
       "triggerID" : "1d621a3f57b64dbbd9f645347267aef8b245ca74",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1d621a3f57b64dbbd9f645347267aef8b245ca74 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9324) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #13988:
URL: https://github.com/apache/flink/pull/13988#issuecomment-723985022


   @rmetzger and @dianfu This is not a new feature, but a conclusion of an API polishing / evolution.
   Getting this into 1.12 would mean we don't have to break the API again in 1.13.
   
   Does this qualify for an exception to be merged after feature freeze?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] sundargates commented on a change in pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
sundargates commented on a change in pull request #13988:
URL: https://github.com/apache/flink/pull/13988#discussion_r520199275



##########
File path: flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumerator.java
##########
@@ -41,12 +43,14 @@
 	void start();
 
 	/**
-	 * Handles the source event from the source reader.
+	 * Handles the request for a split. This method is called when the reader with the given subtask
+	 * id calls the {@link SourceReaderContext#sendSplitRequest()} method.
 	 *
 	 * @param subtaskId the subtask id of the source reader who sent the source event.
-	 * @param sourceEvent the source event from the source reader.
+	 * @param requesterHostname Optional, the hostname where the requesting task is running.
+	 *                          This can be used to make split assignments locality-aware.
 	 */
-	void handleSourceEvent(int subtaskId, SourceEvent sourceEvent);
+	void handleSplitRequest(int subtaskId, @Nullable String requesterHostname);

Review comment:
       +1

##########
File path: flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumerator.java
##########
@@ -87,4 +91,18 @@
 	 */
 	@Override
 	default void notifyCheckpointComplete(long checkpointId) throws Exception {}
+
+	/**
+	 * Handles a custom source event from the source reader.
+	 *
+	 * <p>This method has a default implementation that does nothing, because it is only
+	 * required to be implemented by some sources, which have a custom event protocol between
+	 * reader and enumerator. The common events for reader registration and split requests
+	 * are not dispatched to this method, but rather invoke the {@link #addReader(int)} and
+	 * {@link #handleSplitRequest(int, String)} methods.
+	 *
+	 * @param subtaskId the subtask id of the source reader who sent the source event.
+	 * @param sourceEvent the source event from the source reader.
+	 */
+	default void handleSourceEvent(int subtaskId, SourceEvent sourceEvent) {}

Review comment:
       looks like the enumerators will silently ignore these events if this method was forgotten to be overridden? Wondering if a better default would be to either throw unhandled event exception or at the minimum log them so that the user is aware of the issue? The other thing to keep in mind is that the enumerator will only get custom events if the reader has decided to send them which implies that the user is interested in these events in the first place. 

##########
File path: flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumeratorContext.java
##########
@@ -83,6 +83,14 @@ default void assignSplit(SplitT split, int subtask) {
 		assignSplits(new SplitsAssignment<>(split, subtask));
 	}
 
+	/**
+	 * Signals a subtask that it will not receive any further split.
+	 *
+	 * @param subtask The index of the operator's parallel subtask that shall be
+	 *                signaled it will not receive any further split.
+	 */
+	void signalNoMoreSplits(int subtask);

Review comment:
       Would it also make sense to change this method signature to accept the list of subtasks to be notified? Also, can we have another method that can be used to notify all registered readers?
   
   ```
   void signalNoMoreSplits(Set<Integer> subtasksToBeNotified);
   
   default void signalNoMoreSplits() {
     signalNoMoreSplits(registeredReaders.keySet());
   }
   ```
   
   This is especially useful for bounded cases where you do not know ahead of time if there is any potential that a subtask could be assigned splits in the future based on the current state of the system as assigned splits could always be added back and could be reassigned to other subtasks. Only when the system has reached terminal state (i.e., no more pending splits, all splits completed), you can know for sure all subtasks can be marked as done. So having this method would help in not having to go through the coordinator thread for every single registered subtask. 

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/source/event/NoMoreSplitsEvent.java
##########
@@ -16,16 +16,16 @@
  * limitations under the License.
  */
 
-package org.apache.flink.api.connector.source.event;
+package org.apache.flink.runtime.source.event;
 
-import org.apache.flink.api.connector.source.SourceEvent;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
 
 /**
  * A source event sent from the SplitEnumerator to the SourceReader to indicate that no more
  * splits will be assigned to the source reader anymore. So once the SplitReader finishes
  * reading the currently assigned splits, they can exit.
  */
-public class NoMoreSplitsEvent implements SourceEvent {
+public class NoMoreSplitsEvent implements OperatorEvent {

Review comment:
       nit: should this be a final class?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #13988:
URL: https://github.com/apache/flink/pull/13988#issuecomment-724349363


   Manually merged in 0e281f75386fad5fb2c0c3b10e246b37d02e432d


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rmetzger commented on pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #13988:
URL: https://github.com/apache/flink/pull/13988#issuecomment-723992057


   I'm also fine, since the source API is still marked as beta


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen closed pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
StephanEwen closed pull request #13988:
URL: https://github.com/apache/flink/pull/13988


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13988:
URL: https://github.com/apache/flink/pull/13988#issuecomment-723687968


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 1d621a3f57b64dbbd9f645347267aef8b245ca74 (Mon Nov 09 00:11:40 UTC 2020)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dianfu commented on pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
dianfu commented on pull request #13988:
URL: https://github.com/apache/flink/pull/13988#issuecomment-723986759


   @StephanEwen I tend to think that we should get this into 1.12 to avoid breaking the API again in 1.13. +1 from my side.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] becketqin commented on a change in pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
becketqin commented on a change in pull request #13988:
URL: https://github.com/apache/flink/pull/13988#discussion_r519532305



##########
File path: flink-core/src/main/java/org/apache/flink/api/connector/source/SplitEnumerator.java
##########
@@ -41,12 +43,14 @@
 	void start();
 
 	/**
-	 * Handles the source event from the source reader.
+	 * Handles the request for a split. This method is called when the reader with the given subtask
+	 * id calls the {@link SourceReaderContext#sendSplitRequest()} method.
 	 *
 	 * @param subtaskId the subtask id of the source reader who sent the source event.
-	 * @param sourceEvent the source event from the source reader.
+	 * @param requesterHostname Optional, the hostname where the requesting task is running.
+	 *                          This can be used to make split assignments locality-aware.
 	 */
-	void handleSourceEvent(int subtaskId, SourceEvent sourceEvent);
+	void handleSplitRequest(int subtaskId, @Nullable String requesterHostname);

Review comment:
       Minor: The `requesterHostName` seems not necessary. The implementation can retrieve this information from `SplitEnumeratorContext.registeredReaders().get(subtaskId)` where the hostname can be stored in the `location` field.
   
   Also, it seems that HostName should be a reader property instead of a request property. So maybe there is no need to include that in the `AddSplitsEvent` either.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13988:
URL: https://github.com/apache/flink/pull/13988#issuecomment-723691966


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1d621a3f57b64dbbd9f645347267aef8b245ca74",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1d621a3f57b64dbbd9f645347267aef8b245ca74",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1d621a3f57b64dbbd9f645347267aef8b245ca74 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13988: [FLINK-19265][FLINK-20049][core] Source API final adjustments

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13988:
URL: https://github.com/apache/flink/pull/13988#issuecomment-723691966


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1d621a3f57b64dbbd9f645347267aef8b245ca74",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9324",
       "triggerID" : "1d621a3f57b64dbbd9f645347267aef8b245ca74",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1d621a3f57b64dbbd9f645347267aef8b245ca74 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9324) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org