You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/05/15 16:57:20 UTC

[GitHub] [beam] Abacn opened a new pull request, #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Abacn opened a new pull request, #26697:
URL: https://github.com/apache/beam/pull/26697

   Fixes #25944
   
   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548538658

   run seed job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548553563

   Run Python xlang KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548549080

   Python xlang KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #26697:
URL: https://github.com/apache/beam/pull/26697#discussion_r1199492867


##########
sdks/python/apache_beam/testing/test_utils_test.py:
##########
@@ -95,6 +95,43 @@ def test_cleanup_topics(self):
     pub_client.delete_topic.assert_called_with(topic=topic.name)
 
 
+class LCGeneratorTest(unittest.TestCase):
+  Generator = utils.LCGenerator
+
+  def test_generator_seed_results(self):
+    generator = self.Generator()
+    generator.seed(0)
+    self.assertEqual(generator.next_int(), 4232237)
+
+    generator.seed(1)
+    self.assertEqual(generator.next_int(), -1151252339)
+    self.assertEqual(generator.next_uint(), 3745583449)
+    self.assertAlmostEqual(generator.random_sample(), 0.375548, delta=1e-6)
+    self.assertEqual(generator.rand_bytes(10), b'\xa6\x8fW\xcb\xb1\xa88]dP')
+
+  def test_generator_seed_jdk_results(self):
+    generator = self.Generator()
+    generator.seed_jdk(0)
+    self.assertEqual(generator.next_int(), -1155484576)
+
+    generator.seed_jdk(1)
+    # the first next_int after seed_jdk(1) is close to seed_jdk(0)
+    self.assertEqual(generator.next_int(), -1155869325)

Review Comment:
   just curious (you probably verified this) - does the value still match the JDK value after some large number of iteration? no need to make a test for this.



##########
sdks/python/apache_beam/testing/synthetic_pipeline.py:
##########
@@ -83,7 +91,14 @@ def bytes(self, length):
     return self.getrandbits(length * 8).to_bytes(length, sys.byteorder)
 
 
-Generator = _Random
+def get_generator(seed: Optional[int] = None, algorithm: Optional[str] = None):
+  if algorithm is None or algorithm == 'builtin':
+    return _Random(seed)
+  else:

Review Comment:
   Let's check or assert that algorithm=='lcg'?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548758425

   Run Load Tests Python Combine Dataflow Batch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548775172

   Haven't change Flink load test because they do not use sdk_location but run on the latest sdk container snapshot. Only when this change gets merged into master can see difference on flink load test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26697:
URL: https://github.com/apache/beam/pull/26697#discussion_r1203348852


##########
sdks/python/apache_beam/testing/test_utils_test.py:
##########
@@ -95,6 +95,43 @@ def test_cleanup_topics(self):
     pub_client.delete_topic.assert_called_with(topic=topic.name)
 
 
+class LCGeneratorTest(unittest.TestCase):
+  Generator = utils.LCGenerator
+
+  def test_generator_seed_results(self):
+    generator = self.Generator()
+    generator.seed(0)
+    self.assertEqual(generator.next_int(), 4232237)
+
+    generator.seed(1)
+    self.assertEqual(generator.next_int(), -1151252339)
+    self.assertEqual(generator.next_uint(), 3745583449)
+    self.assertAlmostEqual(generator.random_sample(), 0.375548, delta=1e-6)
+    self.assertEqual(generator.rand_bytes(10), b'\xa6\x8fW\xcb\xb1\xa88]dP')
+
+  def test_generator_seed_jdk_results(self):
+    generator = self.Generator()
+    generator.seed_jdk(0)
+    self.assertEqual(generator.next_int(), -1155484576)
+
+    generator.seed_jdk(1)
+    # the first next_int after seed_jdk(1) is close to seed_jdk(0)
+    self.assertEqual(generator.next_int(), -1155869325)

Review Comment:
   Just did some more test:
   
   Python code:
   
   ```python
   
   from apache_beam.testing.synthetic_pipeline import get_generator
   
   if __name__ == '__main__':
     gen = get_generator(algorithm='lcg')
     gen.seed_jdk(1)
     for i in range(10):
       print(gen.next_int())
   ```
   
   Java code:
   ```java
   
   public static void main(String[] argv) {
     Random rand = new Random(1);
     for (int i = 0; i < 10; ++i) {
       System.out.println(rand.nextInt());
     }
   }
   ```
   
   Both has result
   ```
   -1155869325
   431529176
   1761283695
   1749940626
   892128508
   155629808
   1429008869
   -1465154083
   -138487339
   -1242363800
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548640774

   run seed job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #26697:
URL: https://github.com/apache/beam/pull/26697#discussion_r1203349148


##########
sdks/python/apache_beam/testing/synthetic_pipeline.py:
##########
@@ -83,7 +91,14 @@ def bytes(self, length):
     return self.getrandbits(length * 8).to_bytes(length, sys.byteorder)
 
 
-Generator = _Random
+def get_generator(seed: Optional[int] = None, algorithm: Optional[str] = None):
+  if algorithm is None or algorithm == 'builtin':
+    return _Random(seed)
+  else:

Review Comment:
   added check



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn merged pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #26697:
URL: https://github.com/apache/beam/pull/26697


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548577690

   Run Python TextIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548787791

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @riteshghorse for label python.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548548533

   Run Python TextIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548773814

   Change is effective to load test:
   
   On master (https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/1405/console)
   
   INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_combine_1_runtime Value: 621
   
   Now (https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch_PR/20/console)
   
   INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_combine_1_runtime Value: 352
   
   This is a 43% improvement. Originally _all_ Python load tests actually are testing the performance of SyntheticSource as it costs mosts of the run time.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548758719

   Run Load Tests Python GBK Dataflow Batch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548660711

   Run Load Tests Python Combine Dataflow Batch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548737814

   run seed job


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548577896

   Python xlang KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548602530

   The change is effective in performance tests:
   
   Previous:
   <img width="219" alt="image" src="https://github.com/apache/beam/assets/8010435/851ad031-07ab-434e-8bbc-ccbd871cf97f">
   
   
   Now:
   <img width="213" alt="image" src="https://github.com/apache/beam/assets/8010435/eb12e749-b3e5-4f0a-b516-6ed28a944835">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548553023

   Run SpannerIO Read 2GB Performance Test Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1549830908

   R: @tvalentyn @AnandInguva


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1555367580

   > 
   This is a 43% improvement. Originally all Python load tests actually are testing the performance of SyntheticSource as it costs mosts of the run time.
   
   sounds like we should consider using pregenerated datasets...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1552279236

   > Nice change and blogpost! how does this implementation compare with numpy's generators in terms of performance?
   
   Thanks for taking a look! I also tested numpy's default generator (PCG64) and it seems much slower than builtin
   
   Code:
   
   ```python
   num_loop = 1_000_000
   length = 1024
   
   startt = time.time()
       for idx in range(num_loop):
           gen = np.random.default_rng(idx)
           c = gen.bytes(length)
       print("Use numpy: ", time.time() - startt)
   ```
   
   Example run: Use numpy:  34.7236704826355
   
   For comparison, the cython implementation takes 1 s; builtin takes 10 s.
   
   If I do not set seed every time, the run time is sth like "Use numpy:  10.303484201431274" similar to builtin, but for numpy it seems not a seed method setting seed of an existing generator available


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548583143

   Run SpannerIO Write 2GB Performance Test Python
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548585077

   Run Python xlang KafkaIO Performance Test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548436260

   ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/26697?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#26697](https://app.codecov.io/gh/apache/beam/pull/26697?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (81992ee) into [master](https://app.codecov.io/gh/apache/beam/commit/fc38a47d740cdd6a9f37b53517cb29347ca677df?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (fc38a47) will **increase** coverage by `0.01%`.
   > The diff coverage is `84.78%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #26697      +/-   ##
   ==========================================
   + Coverage   72.05%   72.07%   +0.01%     
   ==========================================
     Files         745      745              
     Lines      101203   101266      +63     
   ==========================================
   + Hits        72926    72983      +57     
   - Misses      26817    26823       +6     
     Partials     1460     1460              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `81.10% <84.78%> (+<0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://app.codecov.io/gh/apache/beam/pull/26697?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...s/python/apache\_beam/testing/synthetic\_pipeline.py](https://app.codecov.io/gh/apache/beam/pull/26697?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | `81.64% <68.18%> (-0.66%)` | :arrow_down: |
   | [sdks/python/apache\_beam/testing/test\_utils.py](https://app.codecov.io/gh/apache/beam/pull/26697?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3V0aWxzLnB5) | `92.43% <100.00%> (+1.91%)` | :arrow_up: |
   
   ... and [47 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/26697/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548217894

   Tested on Dataflow runner, default setting, 1 worker. data size 10M record, each 1 kB:
   
   Current:
   
   <img width="221" alt="image" src="https://github.com/apache/beam/assets/8010435/272d29ac-3ec5-43a8-8701-2f1b92c21e7f">
   
   
   This PR:
   
   <img width="236" alt="image" src="https://github.com/apache/beam/assets/8010435/22cab62c-d0f8-4ff9-b244-821b6ff67e62">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1548662058

   Run Load Tests Python GBK Dataflow Streaming


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1552260349

   Nice change and blogpost! how does this implementation compare with numpy's generators in terms of performance?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #26697: Implement a fast (lcg) random generator for Python SyntheticSource

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26697:
URL: https://github.com/apache/beam/pull/26697#issuecomment-1549832959

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org