You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/11/12 14:12:50 UTC

[GitHub] [beam] scwhittle opened a new pull request #13318: Improve Nexmark performance:

scwhittle opened a new pull request #13318:
URL: https://github.com/apache/beam/pull/13318


   Change Query3 to use stateful processing and timers without
   CoGroupByKey which introduces additional windowing and shuffling. Also use BagState instead of ValueState<List<>>
   
   Change Query5 to avoid unnecessary lists when combining.
   
   Change Query10 to avoid excessive logging.
   
   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](htt
 ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam
 .apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-737461444


   lgtm


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-729093164


   Can you please file a JIRA and put in description?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] scwhittle commented on a change in pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
scwhittle commented on a change in pull request #13318:
URL: https://github.com/apache/beam/pull/13318#discussion_r526017646



##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -115,19 +109,23 @@ public Query3(NexmarkConfiguration configuration) {
                         "OR".equals(person.state)
                             || "ID".equals(person.state)
                             || "CA".equals(person.state)))
+            .apply(

Review comment:
       ditto

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -197,85 +195,60 @@ private JoinDoFn(String name, int maxAuctionsWaitingTime) {
     @ProcessElement
     public void processElement(
         ProcessContext c,
-        @TimerId(PERSON_STATE_EXPIRING) Timer timer,
-        @StateId(PERSON) ValueState<Person> personState,
-        @StateId(AUCTIONS) ValueState<List<Auction>> auctionsState) {
+        @TimerId(STATE_EXPIRING) Timer timer,
+        @StateId(PERSON) @AlwaysFetched ValueState<Person> personState,
+        @StateId(AUCTIONS) BagState<Auction> auctionsState) {
       // We would *almost* implement this by  rewindowing into the global window and
       // running a combiner over the result. The combiner's accumulator would be the
       // state we use below. However, combiners cannot emit intermediate results, thus
-      // we need to wait for the pending ReduceFn API.
+      // we need to wait for the pending ReduceFn API
 
       Person existingPerson = personState.read();
-      if (existingPerson != null) {
-        // We've already seen the new person event for this person id.
-        // We can join with any new auctions on-the-fly without needing any
-        // additional persistent state.
-        for (Auction newAuction : c.element().getValue().getAll(NexmarkQueryUtil.AUCTION_TAG)) {
-          newAuctionCounter.inc();
-          newOldOutputCounter.inc();
-          c.output(KV.of(newAuction, existingPerson));
-        }
-        return;
-      }
-
-      Person theNewPerson = null;
-      for (Person newPerson : c.element().getValue().getAll(NexmarkQueryUtil.PERSON_TAG)) {
-        if (theNewPerson == null) {
-          theNewPerson = newPerson;
+      if (c.element().getValue().newPerson != null) {

Review comment:
       done

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -197,85 +195,60 @@ private JoinDoFn(String name, int maxAuctionsWaitingTime) {
     @ProcessElement
     public void processElement(
         ProcessContext c,
-        @TimerId(PERSON_STATE_EXPIRING) Timer timer,
-        @StateId(PERSON) ValueState<Person> personState,
-        @StateId(AUCTIONS) ValueState<List<Auction>> auctionsState) {
+        @TimerId(STATE_EXPIRING) Timer timer,
+        @StateId(PERSON) @AlwaysFetched ValueState<Person> personState,
+        @StateId(AUCTIONS) BagState<Auction> auctionsState) {
       // We would *almost* implement this by  rewindowing into the global window and
       // running a combiner over the result. The combiner's accumulator would be the
       // state we use below. However, combiners cannot emit intermediate results, thus
-      // we need to wait for the pending ReduceFn API.
+      // we need to wait for the pending ReduceFn API
 
       Person existingPerson = personState.read();
-      if (existingPerson != null) {
-        // We've already seen the new person event for this person id.
-        // We can join with any new auctions on-the-fly without needing any
-        // additional persistent state.
-        for (Auction newAuction : c.element().getValue().getAll(NexmarkQueryUtil.AUCTION_TAG)) {
-          newAuctionCounter.inc();
-          newOldOutputCounter.inc();
-          c.output(KV.of(newAuction, existingPerson));
-        }
-        return;
-      }
-
-      Person theNewPerson = null;
-      for (Person newPerson : c.element().getValue().getAll(NexmarkQueryUtil.PERSON_TAG)) {
-        if (theNewPerson == null) {
-          theNewPerson = newPerson;
+      if (c.element().getValue().newPerson != null) {
+        Person person = c.element().getValue().newPerson;
+        if (existingPerson == null) {
+          newPersonCounter.inc();
+          personState.write(person);
         } else {
-          if (theNewPerson.equals(newPerson)) {
-            LOG.error("Duplicate person {}", theNewPerson);
+          if (person.equals(existingPerson)) {
+            LOG.error("Duplicate person {}", person);
           } else {
-            LOG.error("Conflicting persons {} and {}", theNewPerson, newPerson);
+            LOG.error("Conflicting persons {} and {}", existingPerson, person);
           }
           fatalCounter.inc();
-          continue;
         }
-        newPersonCounter.inc();
         // We've now seen the person for this person id so can flush any
         // pending auctions for the same seller id (an auction is done by only one seller).

Review comment:
       done

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -83,27 +75,29 @@ public Query3(NexmarkConfiguration configuration) {
 
   @Override
   public PCollection<NameCityStateId> expand(PCollection<Event> events) {
-    int numEventsInPane = 30;
-
-    PCollection<Event> eventsWindowed =
-        events.apply(
-            Window.<Event>into(new GlobalWindows())
-                .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(numEventsInPane)))
-                .discardingFiredPanes()
-                .withAllowedLateness(Duration.ZERO));
-    PCollection<KV<Long, Auction>> auctionsBySellerId =
-        eventsWindowed
+    PCollection<KV<Long, Event>> auctionsBySellerId =
+        events
             // Only want the new auction events.
             .apply(NexmarkQueryUtil.JUST_NEW_AUCTIONS)
 
             // We only want auctions in category 10.
             .apply(name + ".InCategory", Filter.by(auction -> auction.category == 10))
 
             // Key auctions by their seller id.
-            .apply("AuctionBySeller", NexmarkQueryUtil.AUCTION_BY_SELLER);
-
-    PCollection<KV<Long, Person>> personsById =
-        eventsWindowed
+            .apply(

Review comment:
       this doesn't work because we also need to change to the union Event type.
   
   Also Auction::seller has the error "Objects is not a functional interface". 
   
   So I kept as is, let me know if you'd rather WithKeys and then some auction->event with auction pardo. If you do, do you have suggestions on fixing the error?  My java is out of date so I'm not sure how to fix that off the top of my head.

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -197,85 +195,60 @@ private JoinDoFn(String name, int maxAuctionsWaitingTime) {
     @ProcessElement
     public void processElement(
         ProcessContext c,

Review comment:
       done

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query5.java
##########
@@ -53,6 +56,48 @@
 public class Query5 extends NexmarkQueryTransform<AuctionCount> {
   private final NexmarkConfiguration configuration;
 
+  public static class TopCombineFn
+      extends AccumulatingCombineFn<KV<Long, Long>, Accum, KV<Long, List<Long>>> {
+    @Override
+    public Accum createAccumulator() {
+      return new Accum();
+    }
+
+    @DefaultCoder(AvroCoder.class)

Review comment:
       done with tweaks

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -164,21 +165,19 @@ public void processElement(ProcessContext c) {
     @StateId(PERSON)
     private static final StateSpec<ValueState<Person>> personSpec = StateSpecs.value(Person.CODER);
 
-    private static final String PERSON_STATE_EXPIRING = "personStateExpiring";
+    private static final String STATE_EXPIRING = "stateExpiring";
 
     @StateId(AUCTIONS)
-    private final StateSpec<ValueState<List<Auction>>> auctionsSpec =
-        StateSpecs.value(ListCoder.of(Auction.CODER));
+    private final StateSpec<BagState<Auction>> auctionsSpec = StateSpecs.bag(Auction.CODER);
 
-    @TimerId(PERSON_STATE_EXPIRING)
-    private final TimerSpec timerSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+    @TimerId(STATE_EXPIRING)

Review comment:
       Processing time is more equivalent to flink background GC, which is why I changed.  The previous code was never gc'ing auctions without persons, which seemed incorrect.  If we want to use an event time timer, it seems that we would need to keep some additional state to keep track of the highest event time of buffered auctions or people, otherwise we could reset timer inappropriately for late events. What do you think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-737539818


   please fix checkstyle errors


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on a change in pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #13318:
URL: https://github.com/apache/beam/pull/13318#discussion_r534432052



##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query5.java
##########
@@ -53,6 +63,79 @@
 public class Query5 extends NexmarkQueryTransform<AuctionCount> {
   private final NexmarkConfiguration configuration;
 
+  public static class TopCombineFn
+      extends AccumulatingCombineFn<KV<Long, Long>, Accum, KV<Long, List<Long>>> {
+    @Override
+    public Accum createAccumulator() {
+      return new Accum();
+    }
+
+    @Override
+    public Coder<Accum> getAccumulatorCoder(
+        @NonNull CoderRegistry registry, @NonNull Coder<KV<Long, Long>> inputCoder) {
+      JavaFieldSchema provider = new JavaFieldSchema();
+      TypeDescriptor<Accum> typeDescriptor = new TypeDescriptor<Accum>() {};
+      provider.schemaFor(typeDescriptor);

Review comment:
       why do you call schemaFor twice?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-737508020


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] scwhittle commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
scwhittle commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-726341731


   Missed specifying a coder, fixed


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] aaltay commented on pull request #13318: [BEAM-11300] Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-747825431


   Could this be merged based on Reuven's LGTM and the passing tests? Any remaining items?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax merged pull request #13318: [BEAM-11300] Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
reuvenlax merged pull request #13318:
URL: https://github.com/apache/beam/pull/13318


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] scwhittle commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
scwhittle commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-737455805


   Fixed
   
   On Wed, Dec 2, 2020 at 8:38 PM reuvenlax <no...@github.com> wrote:
   
   > *@reuvenlax* commented on this pull request.
   > ------------------------------
   >
   > In
   > sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query5.java
   > <https://github.com/apache/beam/pull/13318#discussion_r534432052>:
   >
   > > @@ -53,6 +63,79 @@
   >  public class Query5 extends NexmarkQueryTransform<AuctionCount> {
   >    private final NexmarkConfiguration configuration;
   >
   > +  public static class TopCombineFn
   > +      extends AccumulatingCombineFn<KV<Long, Long>, Accum, KV<Long, List<Long>>> {
   > +    @Override
   > +    public Accum createAccumulator() {
   > +      return new Accum();
   > +    }
   > +
   > +    @Override
   > +    public Coder<Accum> getAccumulatorCoder(
   > +        @NonNull CoderRegistry registry, @NonNull Coder<KV<Long, Long>> inputCoder) {
   > +      JavaFieldSchema provider = new JavaFieldSchema();
   > +      TypeDescriptor<Accum> typeDescriptor = new TypeDescriptor<Accum>() {};
   > +      provider.schemaFor(typeDescriptor);
   >
   > why do you call schemaFor twice?
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/13318#pullrequestreview-543175578>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ABBZZTE7S3TAUSA4OXULTVTSS2JUXANCNFSM4TTKVY2Q>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] scwhittle commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
scwhittle commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-737188487


   R: @reuvenlax PTAL


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] reuvenlax commented on a change in pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #13318:
URL: https://github.com/apache/beam/pull/13318#discussion_r525369407



##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -197,85 +195,60 @@ private JoinDoFn(String name, int maxAuctionsWaitingTime) {
     @ProcessElement
     public void processElement(
         ProcessContext c,

Review comment:
       Instead of ProcessContext just inject @Element and OutputReceiver parameters

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -115,19 +109,23 @@ public Query3(NexmarkConfiguration configuration) {
                         "OR".equals(person.state)
                             || "ID".equals(person.state)
                             || "CA".equals(person.state)))
+            .apply(

Review comment:
       Ditto - simpler to use WithKeys

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -83,27 +75,29 @@ public Query3(NexmarkConfiguration configuration) {
 
   @Override
   public PCollection<NameCityStateId> expand(PCollection<Event> events) {
-    int numEventsInPane = 30;
-
-    PCollection<Event> eventsWindowed =
-        events.apply(
-            Window.<Event>into(new GlobalWindows())
-                .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(numEventsInPane)))
-                .discardingFiredPanes()
-                .withAllowedLateness(Duration.ZERO));
-    PCollection<KV<Long, Auction>> auctionsBySellerId =
-        eventsWindowed
+    PCollection<KV<Long, Event>> auctionsBySellerId =
+        events
             // Only want the new auction events.
             .apply(NexmarkQueryUtil.JUST_NEW_AUCTIONS)
 
             // We only want auctions in category 10.
             .apply(name + ".InCategory", Filter.by(auction -> auction.category == 10))
 
             // Key auctions by their seller id.
-            .apply("AuctionBySeller", NexmarkQueryUtil.AUCTION_BY_SELLER);
-
-    PCollection<KV<Long, Person>> personsById =
-        eventsWindowed
+            .apply(

Review comment:
       Simpler to do
   
   .apply("EventByAuctionSeller", WithKeys.of(Auction::seller).withKeyType(TypeDescriptors.longs()))

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -197,85 +195,60 @@ private JoinDoFn(String name, int maxAuctionsWaitingTime) {
     @ProcessElement
     public void processElement(
         ProcessContext c,
-        @TimerId(PERSON_STATE_EXPIRING) Timer timer,
-        @StateId(PERSON) ValueState<Person> personState,
-        @StateId(AUCTIONS) ValueState<List<Auction>> auctionsState) {
+        @TimerId(STATE_EXPIRING) Timer timer,
+        @StateId(PERSON) @AlwaysFetched ValueState<Person> personState,
+        @StateId(AUCTIONS) BagState<Auction> auctionsState) {
       // We would *almost* implement this by  rewindowing into the global window and
       // running a combiner over the result. The combiner's accumulator would be the
       // state we use below. However, combiners cannot emit intermediate results, thus
-      // we need to wait for the pending ReduceFn API.
+      // we need to wait for the pending ReduceFn API
 
       Person existingPerson = personState.read();
-      if (existingPerson != null) {
-        // We've already seen the new person event for this person id.
-        // We can join with any new auctions on-the-fly without needing any
-        // additional persistent state.
-        for (Auction newAuction : c.element().getValue().getAll(NexmarkQueryUtil.AUCTION_TAG)) {
-          newAuctionCounter.inc();
-          newOldOutputCounter.inc();
-          c.output(KV.of(newAuction, existingPerson));
-        }
-        return;
-      }
-
-      Person theNewPerson = null;
-      for (Person newPerson : c.element().getValue().getAll(NexmarkQueryUtil.PERSON_TAG)) {
-        if (theNewPerson == null) {
-          theNewPerson = newPerson;
+      if (c.element().getValue().newPerson != null) {

Review comment:
       Comment - this is because Event is a union object, etc.

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query5.java
##########
@@ -53,6 +56,48 @@
 public class Query5 extends NexmarkQueryTransform<AuctionCount> {
   private final NexmarkConfiguration configuration;
 
+  public static class TopCombineFn
+      extends AccumulatingCombineFn<KV<Long, Long>, Accum, KV<Long, List<Long>>> {
+    @Override
+    public Accum createAccumulator() {
+      return new Accum();
+    }
+
+    @DefaultCoder(AvroCoder.class)

Review comment:
       AvroCoder tends to be very inefficient. Either write a custom coder or use SchemaCoder. To use SchemaCoder return the following from getAccumulatorCoder (unfortunately it appears that combiners have not been integrated yet with the SchemaRegistry, as otherwise it would be much simpler - just annotating with @DefaultSchema)
   
   JavaFieldSchema provider = new JavaFieldSchema();
   TypeDescriptor<Accum> typeDescriptor = new TypeDescriptor<Accum>{}();
   return SchemaCoder.of(provider.schemaFor(typeDescriptor), provider.toRowFunction(typeDescriptor), provider.fromRowFunction(typeDescriptor));
   

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -197,85 +195,60 @@ private JoinDoFn(String name, int maxAuctionsWaitingTime) {
     @ProcessElement
     public void processElement(
         ProcessContext c,
-        @TimerId(PERSON_STATE_EXPIRING) Timer timer,
-        @StateId(PERSON) ValueState<Person> personState,
-        @StateId(AUCTIONS) ValueState<List<Auction>> auctionsState) {
+        @TimerId(STATE_EXPIRING) Timer timer,
+        @StateId(PERSON) @AlwaysFetched ValueState<Person> personState,
+        @StateId(AUCTIONS) BagState<Auction> auctionsState) {
       // We would *almost* implement this by  rewindowing into the global window and
       // running a combiner over the result. The combiner's accumulator would be the
       // state we use below. However, combiners cannot emit intermediate results, thus
-      // we need to wait for the pending ReduceFn API.
+      // we need to wait for the pending ReduceFn API
 
       Person existingPerson = personState.read();
-      if (existingPerson != null) {
-        // We've already seen the new person event for this person id.
-        // We can join with any new auctions on-the-fly without needing any
-        // additional persistent state.
-        for (Auction newAuction : c.element().getValue().getAll(NexmarkQueryUtil.AUCTION_TAG)) {
-          newAuctionCounter.inc();
-          newOldOutputCounter.inc();
-          c.output(KV.of(newAuction, existingPerson));
-        }
-        return;
-      }
-
-      Person theNewPerson = null;
-      for (Person newPerson : c.element().getValue().getAll(NexmarkQueryUtil.PERSON_TAG)) {
-        if (theNewPerson == null) {
-          theNewPerson = newPerson;
+      if (c.element().getValue().newPerson != null) {
+        Person person = c.element().getValue().newPerson;
+        if (existingPerson == null) {
+          newPersonCounter.inc();
+          personState.write(person);
         } else {
-          if (theNewPerson.equals(newPerson)) {
-            LOG.error("Duplicate person {}", theNewPerson);
+          if (person.equals(existingPerson)) {
+            LOG.error("Duplicate person {}", person);
           } else {
-            LOG.error("Conflicting persons {} and {}", theNewPerson, newPerson);
+            LOG.error("Conflicting persons {} and {}", existingPerson, person);
           }
           fatalCounter.inc();
-          continue;
         }
-        newPersonCounter.inc();
         // We've now seen the person for this person id so can flush any
         // pending auctions for the same seller id (an auction is done by only one seller).

Review comment:
       If this is not the first time this person has been seen, there's no point in even fetching auctionsState, right?

##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -164,21 +165,19 @@ public void processElement(ProcessContext c) {
     @StateId(PERSON)
     private static final StateSpec<ValueState<Person>> personSpec = StateSpecs.value(Person.CODER);
 
-    private static final String PERSON_STATE_EXPIRING = "personStateExpiring";
+    private static final String STATE_EXPIRING = "stateExpiring";
 
     @StateId(AUCTIONS)
-    private final StateSpec<ValueState<List<Auction>>> auctionsSpec =
-        StateSpecs.value(ListCoder.of(Auction.CODER));
+    private final StateSpec<BagState<Auction>> auctionsSpec = StateSpecs.bag(Auction.CODER);
 
-    @TimerId(PERSON_STATE_EXPIRING)
-    private final TimerSpec timerSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+    @TimerId(STATE_EXPIRING)

Review comment:
       Why are you making the expiration timer a processing-time timer? Isn't event time more correct here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] scwhittle commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
scwhittle commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-737903643


   Fixed


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] scwhittle commented on a change in pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
scwhittle commented on a change in pull request #13318:
URL: https://github.com/apache/beam/pull/13318#discussion_r534115800



##########
File path: sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3.java
##########
@@ -164,21 +165,19 @@ public void processElement(ProcessContext c) {
     @StateId(PERSON)
     private static final StateSpec<ValueState<Person>> personSpec = StateSpecs.value(Person.CODER);
 
-    private static final String PERSON_STATE_EXPIRING = "personStateExpiring";
+    private static final String STATE_EXPIRING = "stateExpiring";
 
     @StateId(AUCTIONS)
-    private final StateSpec<ValueState<List<Auction>>> auctionsSpec =
-        StateSpecs.value(ListCoder.of(Auction.CODER));
+    private final StateSpec<BagState<Auction>> auctionsSpec = StateSpecs.bag(Auction.CODER);
 
-    @TimerId(PERSON_STATE_EXPIRING)
-    private final TimerSpec timerSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
+    @TimerId(STATE_EXPIRING)

Review comment:
       Changed back to event time timer.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] scwhittle commented on pull request #13318: Improve Nexmark performance:

Posted by GitBox <gi...@apache.org>.
scwhittle commented on pull request #13318:
URL: https://github.com/apache/beam/pull/13318#issuecomment-726102760


   R: @reuvenlax 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org