You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "nbali (via GitHub)" <gi...@apache.org> on 2023/09/26 02:05:07 UTC

[GitHub] [beam] nbali commented on issue #26395: [Bug]: Possibly unnecessary prefetch during GroupIntoBatches

nbali commented on issue #26395:
URL: https://github.com/apache/beam/issues/26395#issuecomment-1734717553

   @Abacn 
   Well, I only had time now at work. At least some. I haven't tried running it with the modified runner yet, only just by modifying parameters.
   
   First of all I'm not saying your tests are invalid in any way, but I also did mine and saw huge differences. I didn't mention originally as I thought it's irrelevant and should happen with every `GIB`- my bad - but where I noticed this is the `GroupIntoBatches` inside `BigQueryIO`. More specifically this one:
   https://github.com/apache/beam/blob/bb044f4863cd151214f61174ac3c097bea724098/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java#L686-L689
   
   This is what I use on the BQ.Write:
   ```java
   .withMethod(BigQueryIO.Write.Method.FILE_LOADS)
   .withTriggeringFrequency(...)
   .withAutoSharding()
   ```
   
   For the benchmark I removed some irrelevant parts from my pipeline, but kept the core logic of Kafka.Read->BigQuery.Write. First deployed a flex template, then launched the template with different configurations to start consuming.
   
   About the configurations:
   * the size is hardcoded, so that is *500'000* in every test.
   * the byteSize is configurable, I tested with *2 MB* and *100 MB*.
   * the bufferingDuration is configurable as well, I tested with *1 minute* and *1 hour*.
   
   That gives me 4 scenario.
   
   The source is a kafka stream with 12 partitions, about 1k elements/sec and 1 MB/sec for every partition.
   
   Reaching the prefetchSize (500'000 / 5) takes typically almost 2 minutes, so if we trigger the batches more often we never call the "prefetch" code. The 2 MB limit most likely triggers it immediately in a few seconds, so we should never "prefetch". The 100 MB limit most likely triggers 1 prefetch (100MB / 1KB ~100'000).
   
   After 22 min of runtime:
   - read from kafka: 13-15 GB, 17 million
   - streamed data:
   2 MB + 1 min -> 30,77 GB (*~2)
   2 MB + 1 hour -> 30,99 GB (*~2)
   100 MB + 1 min -> 63,38 GB (*~4)
   100 MB + 1 hour -> 65,09 GB (*~4)
   - `GroupIntoBatches.WithShardedKey` metrics from 18 million element input:
   2 MB + 1 min ->~7000 batches 
   2 MB + 1 hour -> ~7000 batches  
   100 MB + 1 min -> ~700 batches
   100 MB + 1 hour -> ~120 batches
   
   As you can see there are huge differences in the results, but also compared to your test pipeline in the code as well, for example you don't use `.withShardedKey()` and trigger it at a much higher frequency.
   
   For the next run I have increased the 100 MB option to 256 MB to trigger prefetching more (and also switched to `n1-highmem-8` from `n1-standard-8` just to be sure avoid an OOME). Everything else is the same.
   
   Drained after 40 min:
   - read from kafka: 16-19 GB, 20-21 million
   - streamed data: 
   2 MB + 1 min -> 39 GB (*~2)
   2 MB + 1 hour -> 39 GB (*~2)
   256 MB + 1 min -> 75 GB (*~4)
   256 MB + 1 hour -> 96 GB (*~5)
   
   I could only _guess_ the reason you couldn't see it is that you fire too quickly and it might be still cached? I would modify the rate from 100'000/sec to 10'000/sec for example (so with the key creation, it will be 1'000/sec for every key on average).
   
   When I have time again, I will try with a modified runner.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org