You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2018/02/16 10:35:25 UTC

[GitHub] flink pull request #5504: [FLINK-8670] Make MetricRegistryImpl#shutdown non ...

GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/5504

    [FLINK-8670] Make MetricRegistryImpl#shutdown non blocking

    ## What is the purpose of the change
    
    This commit makes the MetricRegistryImpl#shutdown method non blocking. Instead
    of waiting for the completion of the shutdown procedure, the method returns a
    future which is completed once the metric registry has completed the shut down.
    
    This PR is based on #5503.
    
    ## Verifying this change
    
    This change is already covered by existing tests.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink metricRegistryShutDown

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5504.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5504
    
----
commit 51bb532d2490968635214016599475f91c5f97e5
Author: Till Rohrmann <tr...@...>
Date:   2018-02-16T09:08:04Z

    [FLINK-8669] Add completeAll and runAfterwards(Async) to FutureUtils
    
    FutureUtils#completeAll(Collection) takes a collection of futures and returns
    a future which is completed after all of the given futures are completed. This
    also includes exceptional completions. Potentially occurring exceptions are
    recorded and combined into a single exception with which the resulting future
    is completed.
    
    FutureUtils#runAfterwards takes a future and runs a given action after the
    completion of the given future. This also includes an exceptional completion.
    In this case, a potentially occurring exception as the result of the provided
    action will be combined with the future's exception.

commit 45854d259a5ff0d03fe69a9caabc5c8627be3ba9
Author: Till Rohrmann <tr...@...>
Date:   2018-02-16T07:12:04Z

    [FLINK-8670] Make MetricRegistryImpl#shutdown non blocking
    
    This commit makes the MetricRegistryImpl#shutdown method non blocking. Instead
    of waiting for the completion of the shutdown procedure, the method returns a
    future which is completed once the metric registry has completed the shut down.

----


---

[GitHub] flink pull request #5504: [FLINK-8670] Make MetricRegistryImpl#shutdown non ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/5504


---

[GitHub] flink pull request #5504: [FLINK-8670] Make MetricRegistryImpl#shutdown non ...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5504#discussion_r168749048
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java ---
    @@ -226,71 +235,72 @@ public int getNumberReporters() {
     	 */
     	public boolean isShutdown() {
     		synchronized (lock) {
    -			return reporters == null && executor.isShutdown();
    +			return isShutdown;
     		}
     	}
     
     	/**
     	 * Shuts down this registry and the associated {@link MetricReporter}.
    +	 *
    +	 * <p>NOTE: This operation is asynchronous and returns a future which is completed
    +	 * once the shutdown operation has been completed.
    +	 *
    +	 * @return Future which is completed once the {@link MetricRegistryImpl}
    +	 * is shut down.
     	 */
    -	public void shutdown() {
    +	public CompletableFuture<Void> shutdown() {
    --- End diff --
    
    Will do.


---

[GitHub] flink pull request #5504: [FLINK-8670] Make MetricRegistryImpl#shutdown non ...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5504#discussion_r168737362
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java ---
    @@ -226,71 +235,72 @@ public int getNumberReporters() {
     	 */
     	public boolean isShutdown() {
     		synchronized (lock) {
    -			return reporters == null && executor.isShutdown();
    +			return isShutdown;
     		}
     	}
     
     	/**
     	 * Shuts down this registry and the associated {@link MetricReporter}.
    +	 *
    +	 * <p>NOTE: This operation is asynchronous and returns a future which is completed
    +	 * once the shutdown operation has been completed.
    +	 *
    +	 * @return Future which is completed once the {@link MetricRegistryImpl}
    +	 * is shut down.
     	 */
    -	public void shutdown() {
    +	public CompletableFuture<Void> shutdown() {
    --- End diff --
    
    Fair point. What about the renaming though?


---

[GitHub] flink pull request #5504: [FLINK-8670] Make MetricRegistryImpl#shutdown non ...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5504#discussion_r168728890
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java ---
    @@ -226,71 +235,72 @@ public int getNumberReporters() {
     	 */
     	public boolean isShutdown() {
     		synchronized (lock) {
    -			return reporters == null && executor.isShutdown();
    +			return isShutdown;
     		}
     	}
     
     	/**
     	 * Shuts down this registry and the associated {@link MetricReporter}.
    +	 *
    +	 * <p>NOTE: This operation is asynchronous and returns a future which is completed
    +	 * once the shutdown operation has been completed.
    +	 *
    +	 * @return Future which is completed once the {@link MetricRegistryImpl}
    +	 * is shut down.
     	 */
    -	public void shutdown() {
    +	public CompletableFuture<Void> shutdown() {
    --- End diff --
    
    How about renaming this to shutdownAsync? (and possible keep shutdown() that just calls shutdownAsync().get()).


---

[GitHub] flink pull request #5504: [FLINK-8670] Make MetricRegistryImpl#shutdown non ...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5504#discussion_r168735349
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java ---
    @@ -77,19 +85,20 @@
     
     	private ViewUpdater viewUpdater;
     
    -	private final ScopeFormats scopeFormats;
    -	private final char globalDelimiter;
    -	private final List<Character> delimiters = new ArrayList<>();
    +	private boolean isShutdown;
     
     	/**
     	 * Creates a new MetricRegistry and starts the configured reporter.
     	 */
     	public MetricRegistryImpl(MetricRegistryConfiguration config) {
     		this.scopeFormats = config.getScopeFormats();
     		this.globalDelimiter = config.getDelimiter();
    +		this.delimiters = new ArrayList<>(10);
    --- End diff --
    
    Good point, will change it.


---

[GitHub] flink pull request #5504: [FLINK-8670] Make MetricRegistryImpl#shutdown non ...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5504#discussion_r168735497
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java ---
    @@ -226,71 +235,72 @@ public int getNumberReporters() {
     	 */
     	public boolean isShutdown() {
     		synchronized (lock) {
    -			return reporters == null && executor.isShutdown();
    +			return isShutdown;
     		}
     	}
     
     	/**
     	 * Shuts down this registry and the associated {@link MetricReporter}.
    +	 *
    +	 * <p>NOTE: This operation is asynchronous and returns a future which is completed
    +	 * once the shutdown operation has been completed.
    +	 *
    +	 * @return Future which is completed once the {@link MetricRegistryImpl}
    +	 * is shut down.
     	 */
    -	public void shutdown() {
    +	public CompletableFuture<Void> shutdown() {
    --- End diff --
    
    Hmm, I actually would like to avoid that people can introduce blocking operations into the `ClusterEntrypoint`, for example. By offering a blocking `shutdown` method, I fear that this can easily happen again.


---

[GitHub] flink pull request #5504: [FLINK-8670] Make MetricRegistryImpl#shutdown non ...

Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5504#discussion_r168728354
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java ---
    @@ -77,19 +85,20 @@
     
     	private ViewUpdater viewUpdater;
     
    -	private final ScopeFormats scopeFormats;
    -	private final char globalDelimiter;
    -	private final List<Character> delimiters = new ArrayList<>();
    +	private boolean isShutdown;
     
     	/**
     	 * Creates a new MetricRegistry and starts the configured reporter.
     	 */
     	public MetricRegistryImpl(MetricRegistryConfiguration config) {
     		this.scopeFormats = config.getScopeFormats();
     		this.globalDelimiter = config.getDelimiter();
    +		this.delimiters = new ArrayList<>(10);
    --- End diff --
    
    the number of delimiters is always equal to the number of reporters, so they should be initialized with the same size.


---