You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/12 20:49:02 UTC

[GitHub] [beam] robertwb opened a new pull request, #23194: Batch encoding and decoding of schema data.

robertwb opened a new pull request, #23194:
URL: https://github.com/apache/beam/pull/23194

   This implements conversion directly from a dict of numpy arrays to a concatinated-row-encoded form, and vice versa.
   
   This is generally 5-20x more efficient for encoding and 2-10x more efficient for decoding, depending on the row types and contents.
   
       Benchmarks:
       
       tiny_row, BatchRowCoderFalse, 1000 element(s)    : p. element median time cost: 3.79658e-06 sec, relative std: 5.82%
       tiny_row, BatchRowCoderTrue, 1000 element(s)     : p. element median time cost: 3.9053e-07 sec, relative std: 6256.41%
       large_row, BatchRowCoderFalse, 1000 element(s)   : p. element median time cost: 2.4623e-05 sec, relative std: 2.46%
       large_row, BatchRowCoderTrue, 1000 element(s)    : p. element median time cost: 1.91796e-06 sec, relative std: 8.61%
       nullable_row, BatchRowCoderFalse, 1000 element(s): p. element median time cost: 2.3849e-05 sec, relative std: 2.15%
       nullable_row, BatchRowCoderTrue, 1000 element(s) : p. element median time cost: 2.27439e-06 sec, relative std: 28.73%
       diverse_row, BatchRowCoderFalse, 1000 element(s) : p. element median time cost: 9.17304e-06 sec, relative std: 2.60%
       diverse_row, BatchRowCoderTrue, 1000 element(s)  : p. element median time cost: 2.42209e-06 sec, relative std: 10.16%
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] robertwb commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
robertwb commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1248828133

   The errors look real--looking into them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] robertwb commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
robertwb commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1263915755

   Note also that windowing still needs to be taken into account, but with the value only coder this code can still be used directly. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] robertwb commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
robertwb commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1245982201

   On Tue, Sep 13, 2022 at 12:05 PM Brian Hulette ***@***.***>
   wrote:
   
   > Very cool, thank you! I will take a detailed look in the next few days.
   > First some high-level comments:
   >
   >    - This currently isn't used anywhere, right? Any thoughts on how we
   >    would plumb this in?
   >
   > Correct, this isn't used yet. I was thinking we could possibly add methods
   like "encode_batch" and "decode_batch" to the BatchConverters that would
   take a coder, whose default implementation would be to explode/encode (or
   vice versa) but could be overridden. The input/output ports would take
   advantage of this.
   
   >
   >    - I think it would be a good idea to exercise this with the
   >    integration tests in standard_coders.yaml. In a few cases we have multiple
   >    test values (e.g. here:
   >    https://github.com/apache/beam/blob/6f25fba3fea1f407c7a008c84709047b7a55be54/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml#L357-L364).
   >    We could concatenate together the encoded representations and verify a
   >    batch decode.
   >
   >
   Good idea. Will do.
   
   
   >
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/23194#issuecomment-1245842369>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AADWVAP7DOB3LYJSMJZOBQLV6DF6DANCNFSM6AAAAAAQK2LWFU>
   > .
   > You are receiving this because you authored the thread.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] robertwb commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
robertwb commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1244450205

   R: @TheNeuralBit 
   
   We could use this for the input/output ports over the FnAPI boundary. For now it'd mostly help with cross-language, but in the future this (or a variant) could be useful for the GBK crossings as well. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] TheNeuralBit commented on a diff in pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on code in PR #23194:
URL: https://github.com/apache/beam/pull/23194#discussion_r984785755


##########
sdks/python/apache_beam/coders/coder_impl_row_encoders.pyx:
##########
@@ -0,0 +1,168 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Optimized implementations of various schema columner types."""
+
+# pytype: skip-file
+
+import numpy as np
+cimport numpy as np
+
+from apache_beam.coders import coder_impl
+from apache_beam.coders.coder_impl cimport RowColumnEncoder, OutputStream, InputStream
+from apache_beam.portability.api import schema_pb2
+
+
+cdef class AtomicTypeRowColumnEncoder(RowColumnEncoder):
+  cdef original
+  cdef contiguous
+
+  def __init__(self, column):
+    self.original = column
+    self.contiguous = np.ascontiguousarray(column)
+
+  def null_flags(self):
+    return None
+
+  def finalize_write(self):
+    if self.original is not self.contiguous:
+      self.original[:] = self.contiguous
+
+
+cdef class FloatFloat32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(FloatFloat32RowColumnEncoder, self).__init__(column)
+    cdef np.float32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_float(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_float()
+
+FloatFloat32RowColumnEncoder.register(schema_pb2.FLOAT, np.float32().dtype)
+
+
+cdef class FloatFloat64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(FloatFloat64RowColumnEncoder, self).__init__(column)
+    cdef np.float64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_float(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_float()
+
+FloatFloat64RowColumnEncoder.register(schema_pb2.FLOAT, np.float64().dtype)
+
+
+cdef class DoubleFloat32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(DoubleFloat32RowColumnEncoder, self).__init__(column)
+    cdef np.float32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_double(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_double()
+
+DoubleFloat32RowColumnEncoder.register(schema_pb2.DOUBLE, np.float32().dtype)
+
+
+cdef class DoubleFloat64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(DoubleFloat64RowColumnEncoder, self).__init__(column)
+    cdef np.float64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_double(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_double()
+
+DoubleFloat64RowColumnEncoder.register(schema_pb2.DOUBLE, np.float64().dtype)
+
+
+cdef class Int32Int32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.int32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(Int32Int32RowColumnEncoder, self).__init__(column)
+    cdef np.int32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_var_int64(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_var_int64()
+
+Int32Int32RowColumnEncoder.register(schema_pb2.INT32, np.int32().dtype)
+Int32Int32RowColumnEncoder.register(schema_pb2.INT32, np.int64().dtype)
+
+
+cdef class Int64Int64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.int64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(Int64Int64RowColumnEncoder, self).__init__(column)
+    cdef np.int64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_var_int64(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_var_int64()
+
+Int64Int64RowColumnEncoder.register(schema_pb2.INT64, np.int64().dtype)
+
+
+cdef class BoolRowColumnEncoder(AtomicTypeRowColumnEncoder):

Review Comment:
   nit: ByteRowColumnEncoder?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] codecov[bot] commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1244717173

   # [Codecov](https://codecov.io/gh/apache/beam/pull/23194?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#23194](https://codecov.io/gh/apache/beam/pull/23194?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1925d8) into [master](https://codecov.io/gh/apache/beam/commit/70f8728cd665c00855199282995f49ca08351db7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (70f8728) will **increase** coverage by `0.02%`.
   > The diff coverage is `91.30%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #23194      +/-   ##
   ==========================================
   + Coverage   73.58%   73.60%   +0.02%     
   ==========================================
     Files         716      716              
     Lines       95301    95398      +97     
   ==========================================
   + Hits        70125    70216      +91     
   - Misses      23880    23886       +6     
     Partials     1296     1296              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `83.41% <91.30%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/23194?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyX2ltcGwucHk=) | `93.61% <91.30%> (-0.30%)` | :arrow_down: |
   | [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `95.12% <0.00%> (-2.44%)` | :arrow_down: |
   | [sdks/python/apache\_beam/internal/metrics/metric.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW50ZXJuYWwvbWV0cmljcy9tZXRyaWMucHk=) | `93.00% <0.00%> (-1.00%)` | :arrow_down: |
   | [...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==) | `92.44% <0.00%> (-0.65%)` | :arrow_down: |
   | [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `91.71% <0.00%> (-0.31%)` | :arrow_down: |
   | [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.59% <0.00%> (-0.13%)` | :arrow_down: |
   | [setup.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2V0dXAucHk=) | `0.00% <0.00%> (ø)` | |
   | [...thon/apache\_beam/ml/inference/pytorch\_inference.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL3B5dG9yY2hfaW5mZXJlbmNlLnB5) | `0.00% <0.00%> (ø)` | |
   | [...beam/runners/portability/expansion\_service\_main.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9leHBhbnNpb25fc2VydmljZV9tYWluLnB5) | `0.00% <0.00%> (ø)` | |
   | [...am/examples/inference/pytorch\_language\_modeling.py](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3B5dG9yY2hfbGFuZ3VhZ2VfbW9kZWxpbmcucHk=) | `0.00% <0.00%> (ø)` | |
   | ... and [9 more](https://codecov.io/gh/apache/beam/pull/23194/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] TheNeuralBit commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1245842369

   Very cool, thank you! I will take a detailed look in the next few days. First some high-level comments:
   - This currently isn't used anywhere, right? Any thoughts on how we would plumb this in?
   - I think it would be a good idea to exercise this with the integration tests in standard_coders.yaml. In a few cases we have multiple test values (e.g. here: https://github.com/apache/beam/blob/6f25fba3fea1f407c7a008c84709047b7a55be54/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml#L357-L364). We could concatenate together the encoded representations and verify a batch decode.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] robertwb merged pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
robertwb merged PR #23194:
URL: https://github.com/apache/beam/pull/23194


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] TheNeuralBit commented on a diff in pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on code in PR #23194:
URL: https://github.com/apache/beam/pull/23194#discussion_r984977942


##########
sdks/python/apache_beam/coders/coder_impl_row_encoders.pyx:
##########
@@ -0,0 +1,168 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Optimized implementations of various schema columner types."""
+
+# pytype: skip-file
+
+import numpy as np
+cimport numpy as np
+
+from apache_beam.coders import coder_impl
+from apache_beam.coders.coder_impl cimport RowColumnEncoder, OutputStream, InputStream
+from apache_beam.portability.api import schema_pb2
+
+
+cdef class AtomicTypeRowColumnEncoder(RowColumnEncoder):
+  cdef original
+  cdef contiguous
+
+  def __init__(self, column):
+    self.original = column
+    self.contiguous = np.ascontiguousarray(column)
+
+  def null_flags(self):
+    return None
+
+  def finalize_write(self):
+    if self.original is not self.contiguous:
+      self.original[:] = self.contiguous
+
+
+cdef class FloatFloat32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(FloatFloat32RowColumnEncoder, self).__init__(column)
+    cdef np.float32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_float(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_float()
+
+FloatFloat32RowColumnEncoder.register(schema_pb2.FLOAT, np.float32().dtype)
+
+
+cdef class FloatFloat64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(FloatFloat64RowColumnEncoder, self).__init__(column)
+    cdef np.float64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_float(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_float()
+
+FloatFloat64RowColumnEncoder.register(schema_pb2.FLOAT, np.float64().dtype)
+
+
+cdef class DoubleFloat32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(DoubleFloat32RowColumnEncoder, self).__init__(column)
+    cdef np.float32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_double(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_double()
+
+DoubleFloat32RowColumnEncoder.register(schema_pb2.DOUBLE, np.float32().dtype)
+
+
+cdef class DoubleFloat64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(DoubleFloat64RowColumnEncoder, self).__init__(column)
+    cdef np.float64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_double(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_double()
+
+DoubleFloat64RowColumnEncoder.register(schema_pb2.DOUBLE, np.float64().dtype)
+
+
+cdef class Int32Int32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.int32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(Int32Int32RowColumnEncoder, self).__init__(column)
+    cdef np.int32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_var_int64(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_var_int64()
+
+Int32Int32RowColumnEncoder.register(schema_pb2.INT32, np.int32().dtype)
+Int32Int32RowColumnEncoder.register(schema_pb2.INT32, np.int64().dtype)
+
+
+cdef class Int64Int64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.int64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(Int64Int64RowColumnEncoder, self).__init__(column)
+    cdef np.int64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_var_int64(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_var_int64()
+
+Int64Int64RowColumnEncoder.register(schema_pb2.INT64, np.int64().dtype)
+
+
+cdef class BoolRowColumnEncoder(AtomicTypeRowColumnEncoder):

Review Comment:
   Oh I see, I was confused by the np.int8 and np.uint8 registrations below, but I see now that's a sort of shunt.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] TheNeuralBit commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1263696973

   Run Python PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] robertwb commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
robertwb commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1263914988

   > I wonder if we should prefer arrow arrays as the default column type though? That way we can specialize strings as well. I think this is OK as-is though since it's not used anywhere and it's effectively a proof of concept. We can experiment with arrow/strings later.
   
   I think numpy is a lower common denominator than arrow, but this framework certainly lends itself to registration of column encoders of various types, including ragged, justified, or packed arrow string types. (Technically this currently works for anything supporting the https://docs.python.org/3/c-api/buffer.html , though an explicit lookup on the dtype would need to be generalize.)
   
   And, yes, the primary intent is when batch DoFns are adjacent to cross-SDK boundaries of some type. 
   
   Currently, I don't think there's room for savings in decoding a batch + exploding over decoding individual rows (or vice versa) but the former could be a win if we have a row representation that's simply an index into a batch (though that might simply push some of the cost to attribute access).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] TheNeuralBit commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1263875515

   FYI: @apilloud and @chamikaramj
   
   Currently this is just theoretical, but with some plumbing we could realize a benefit from this when a Batched DoFn is adjacent to a cross-language transform that uses RowCoder, or a shuffle boundary.
   
   @robertwb: I wonder if it would also be faster to use this even in the row-oriented case? Have you checked the performance of this + batching/unbatching against the standard RowCoder?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] robertwb commented on a diff in pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
robertwb commented on code in PR #23194:
URL: https://github.com/apache/beam/pull/23194#discussion_r984865333


##########
sdks/python/apache_beam/coders/coder_impl_row_encoders.pyx:
##########
@@ -0,0 +1,168 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Optimized implementations of various schema columner types."""
+
+# pytype: skip-file
+
+import numpy as np
+cimport numpy as np
+
+from apache_beam.coders import coder_impl
+from apache_beam.coders.coder_impl cimport RowColumnEncoder, OutputStream, InputStream
+from apache_beam.portability.api import schema_pb2
+
+
+cdef class AtomicTypeRowColumnEncoder(RowColumnEncoder):
+  cdef original
+  cdef contiguous
+
+  def __init__(self, column):
+    self.original = column
+    self.contiguous = np.ascontiguousarray(column)
+
+  def null_flags(self):
+    return None
+
+  def finalize_write(self):
+    if self.original is not self.contiguous:
+      self.original[:] = self.contiguous
+
+
+cdef class FloatFloat32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(FloatFloat32RowColumnEncoder, self).__init__(column)
+    cdef np.float32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_float(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_float()
+
+FloatFloat32RowColumnEncoder.register(schema_pb2.FLOAT, np.float32().dtype)
+
+
+cdef class FloatFloat64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(FloatFloat64RowColumnEncoder, self).__init__(column)
+    cdef np.float64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_float(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_float()
+
+FloatFloat64RowColumnEncoder.register(schema_pb2.FLOAT, np.float64().dtype)
+
+
+cdef class DoubleFloat32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(DoubleFloat32RowColumnEncoder, self).__init__(column)
+    cdef np.float32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_double(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_double()
+
+DoubleFloat32RowColumnEncoder.register(schema_pb2.DOUBLE, np.float32().dtype)
+
+
+cdef class DoubleFloat64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.float64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(DoubleFloat64RowColumnEncoder, self).__init__(column)
+    cdef np.float64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_bigendian_double(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_bigendian_double()
+
+DoubleFloat64RowColumnEncoder.register(schema_pb2.DOUBLE, np.float64().dtype)
+
+
+cdef class Int32Int32RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.int32_t* data
+
+  def __init__(self, unused_coder, column):
+    super(Int32Int32RowColumnEncoder, self).__init__(column)
+    cdef np.int32_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_var_int64(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_var_int64()
+
+Int32Int32RowColumnEncoder.register(schema_pb2.INT32, np.int32().dtype)
+Int32Int32RowColumnEncoder.register(schema_pb2.INT32, np.int64().dtype)
+
+
+cdef class Int64Int64RowColumnEncoder(AtomicTypeRowColumnEncoder):
+  cdef np.int64_t* data
+
+  def __init__(self, unused_coder, column):
+    super(Int64Int64RowColumnEncoder, self).__init__(column)
+    cdef np.int64_t[::1] view = self.contiguous
+    self.data = &view[0]
+
+  cdef bint encode_to_stream(self, size_t index, OutputStream stream) except -1:
+    stream.write_var_int64(self.data[index])
+
+  cdef bint decode_from_stream(self, size_t index, InputStream stream) except -1:
+    self.data[index] = stream.read_var_int64()
+
+Int64Int64RowColumnEncoder.register(schema_pb2.INT64, np.int64().dtype)
+
+
+cdef class BoolRowColumnEncoder(AtomicTypeRowColumnEncoder):

Review Comment:
   This is about encoding schema_pb2.BOOLEAN. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #23194: Batch encoding and decoding of schema data.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23194:
URL: https://github.com/apache/beam/pull/23194#issuecomment-1244452481

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org