You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "udim (via GitHub)" <gi...@apache.org> on 2023/06/08 23:14:38 UTC

[GitHub] [beam] udim opened a new pull request, #27078: Initial implementation of QuotaEvent

udim opened a new pull request, #27078:
URL: https://github.com/apache/beam/pull/27078

   Part of #26460
   
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] bvolpato commented on a diff in pull request #27078: Initial implementation of QuotaEvent

Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on code in PR #27078:
URL: https://github.com/apache/beam/pull/27078#discussion_r1227470717


##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/QuotaEvent.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.fn.harness.logging;
+
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.slf4j.MDC;
+
+/**
+ * Adds MDC key:value pairs for a quota event. The contents of the MDC are added to each log entry.
+ *
+ * <p>Suggested usage: try (QuotaEventCloseable qec = new
+ * QuotaEvent.Builder().withOperation(op).withFullResourceName(name).create()) { LOG.info(...); }
+ */
+public class QuotaEvent {
+  public static class QuotaEventCloseable implements AutoCloseable {
+    public QuotaEventCloseable(Map<String, String> data) {
+      this.data = data;
+    }
+
+    @Override
+    public void close() {
+      if (!enabled) {
+        return;
+      }
+      for (String key : data.keySet()) {
+        MDC.remove(key);
+      }

Review Comment:
   Shouldn't we apply back what was in the stack instead of removing? Or we have no expectations for MDCs to be re-entrant?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] bvolpato commented on pull request #27078: Initial implementation of QuotaEvent

Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #27078:
URL: https://github.com/apache/beam/pull/27078#issuecomment-1590327814

   LGTM, thanks Udi!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] udim commented on a diff in pull request #27078: Initial implementation of QuotaEvent

Posted by "udim (via GitHub)" <gi...@apache.org>.
udim commented on code in PR #27078:
URL: https://github.com/apache/beam/pull/27078#discussion_r1228656201


##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/QuotaEvent.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.fn.harness.logging;
+
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.slf4j.MDC;
+
+/**
+ * Adds MDC key:value pairs for a quota event. The contents of the MDC are added to each log entry.
+ *
+ * <p>Suggested usage: try (QuotaEventCloseable qec = new
+ * QuotaEvent.Builder().withOperation(op).withFullResourceName(name).create()) { LOG.info(...); }
+ */
+public class QuotaEvent {
+  public static class QuotaEventCloseable implements AutoCloseable {
+    public QuotaEventCloseable(Map<String, String> data) {
+      this.data = data;
+    }
+
+    @Override
+    public void close() {
+      if (!enabled) {
+        return;
+      }
+      for (String key : data.keySet()) {
+        MDC.remove(key);
+      }

Review Comment:
   PTAL



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] bvolpato commented on pull request #27078: Initial implementation of QuotaEvent

Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #27078:
URL: https://github.com/apache/beam/pull/27078#issuecomment-1588446503

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] udim merged pull request #27078: Initial implementation of QuotaEvent

Posted by "udim (via GitHub)" <gi...@apache.org>.
udim merged PR #27078:
URL: https://github.com/apache/beam/pull/27078


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] udim commented on pull request #27078: Initial implementation of QuotaEvent

Posted by "udim (via GitHub)" <gi...@apache.org>.
udim commented on PR #27078:
URL: https://github.com/apache/beam/pull/27078#issuecomment-1588247975

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] udim commented on pull request #27078: Initial implementation of QuotaEvent

Posted by "udim (via GitHub)" <gi...@apache.org>.
udim commented on PR #27078:
URL: https://github.com/apache/beam/pull/27078#issuecomment-1588198885

   Review ping
   
   Benchmark results don't show any performance degredation when using QuotaEvent:
   ![image](https://github.com/apache/beam/assets/127695/4d4fef7c-0813-40c6-8783-3c1618485cdc)
   
   Command: `./gradlew :sdks:java:harness:jmh:jmh -Pbenchmark=BeamFnLoggingClientBenchmark`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] udim commented on a diff in pull request #27078: Initial implementation of QuotaEvent

Posted by "udim (via GitHub)" <gi...@apache.org>.
udim commented on code in PR #27078:
URL: https://github.com/apache/beam/pull/27078#discussion_r1228543208


##########
sdks/java/harness/jmh/src/main/java/org/apache/beam/fn/harness/jmh/logging/BeamFnLoggingClientBenchmark.java:
##########
@@ -206,6 +207,23 @@ public void testLoggingWithCustomData(
     }
   }
 
+  @Benchmark
+  @Threads(16) // Use several threads since we expect contention during logging
+  public void testLoggingWithQuotaEvent(
+      ManyExpectedCallsLoggingClientAndService client, ManageExecutionState executionState)
+      throws Exception {
+    try (Closeable state =
+        executionState.executionStateTracker.enterState(executionState.simpleExecutionState)) {
+      try (AutoCloseable ac =
+          new QuotaEvent.Builder()
+              .withOperation("test")
+              .withFullResourceName("//test.googleapis.com/abc/123")
+              .create()) {
+        LOG.warn("log me");

Review Comment:
   To be clearer: the SDK's logging handler adds the MDC data to the LogEntry, which is passed to the FnAPI logging handler. The standard JUL (java.util.logging) handler does not use MDC so it's harder to test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #27078: Initial implementation of QuotaEvent

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27078:
URL: https://github.com/apache/beam/pull/27078#issuecomment-1583700747

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @bvolpato for label java.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] udim commented on a diff in pull request #27078: Initial implementation of QuotaEvent

Posted by "udim (via GitHub)" <gi...@apache.org>.
udim commented on code in PR #27078:
URL: https://github.com/apache/beam/pull/27078#discussion_r1228499066


##########
sdks/java/harness/jmh/src/main/java/org/apache/beam/fn/harness/jmh/logging/BeamFnLoggingClientBenchmark.java:
##########
@@ -206,6 +207,23 @@ public void testLoggingWithCustomData(
     }
   }
 
+  @Benchmark
+  @Threads(16) // Use several threads since we expect contention during logging
+  public void testLoggingWithQuotaEvent(
+      ManyExpectedCallsLoggingClientAndService client, ManageExecutionState executionState)
+      throws Exception {
+    try (Closeable state =
+        executionState.executionStateTracker.enterState(executionState.simpleExecutionState)) {
+      try (AutoCloseable ac =
+          new QuotaEvent.Builder()
+              .withOperation("test")
+              .withFullResourceName("//test.googleapis.com/abc/123")
+              .create()) {
+        LOG.warn("log me");

Review Comment:
   I have manually verified that MDC in general (not QuotaEvent) works end-to-end on Dataflow. I plan on adding an integration test, once all the required Dataflow pieces have rolled out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] bvolpato commented on a diff in pull request #27078: Initial implementation of QuotaEvent

Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on code in PR #27078:
URL: https://github.com/apache/beam/pull/27078#discussion_r1227470347


##########
sdks/java/harness/jmh/src/main/java/org/apache/beam/fn/harness/jmh/logging/BeamFnLoggingClientBenchmark.java:
##########
@@ -206,6 +207,23 @@ public void testLoggingWithCustomData(
     }
   }
 
+  @Benchmark
+  @Threads(16) // Use several threads since we expect contention during logging
+  public void testLoggingWithQuotaEvent(
+      ManyExpectedCallsLoggingClientAndService client, ManageExecutionState executionState)
+      throws Exception {
+    try (Closeable state =
+        executionState.executionStateTracker.enterState(executionState.simpleExecutionState)) {
+      try (AutoCloseable ac =
+          new QuotaEvent.Builder()
+              .withOperation("test")
+              .withFullResourceName("//test.googleapis.com/abc/123")
+              .create()) {
+        LOG.warn("log me");

Review Comment:
   Just curious, how do we validate that the proper MDC is in place? Seems like we have benchmarks but it isn't clear if we test the functionality.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] udim commented on a diff in pull request #27078: Initial implementation of QuotaEvent

Posted by "udim (via GitHub)" <gi...@apache.org>.
udim commented on code in PR #27078:
URL: https://github.com/apache/beam/pull/27078#discussion_r1228511750


##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/QuotaEvent.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.fn.harness.logging;
+
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.slf4j.MDC;
+
+/**
+ * Adds MDC key:value pairs for a quota event. The contents of the MDC are added to each log entry.
+ *
+ * <p>Suggested usage: try (QuotaEventCloseable qec = new
+ * QuotaEvent.Builder().withOperation(op).withFullResourceName(name).create()) { LOG.info(...); }
+ */
+public class QuotaEvent {
+  public static class QuotaEventCloseable implements AutoCloseable {
+    public QuotaEventCloseable(Map<String, String> data) {
+      this.data = data;
+    }
+
+    @Override
+    public void close() {
+      if (!enabled) {
+        return;
+      }
+      for (String key : data.keySet()) {
+        MDC.remove(key);
+      }

Review Comment:
   I expect that QuotaEvent's MDC keys are exclusively used by QuotaEvent. I don't expect nested QuotaEvents to happen.
   You're right though that this is unexpected, and should at least be documented.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org