You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/19 21:52:00 UTC

[GitHub] [arrow] zeroshade opened a new pull request, #13929: ARROW-17479: [Go] Add ArraySpan and utilities

zeroshade opened a new pull request, #13929:
URL: https://github.com/apache/arrow/pull/13929

   Relating to the building of the functionality for Compute in Go with Arrow, this is the implementation of ArraySpan / ExecValue / ExecResult etc.
   
   It was able to be separated out from the function interface definitions, so I was able to make this PR while #13924 is still being reviewed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] lidavidm commented on pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #13929:
URL: https://github.com/apache/arrow/pull/13929#issuecomment-1221330378

   Just wondering - would ArraySpan be an idiomatic name in Go? I think the C++ name derives from `std::span`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade commented on pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
zeroshade commented on PR #13929:
URL: https://github.com/apache/arrow/pull/13929#issuecomment-1221325679

   @pitrou yup I'm aware. I haven't done the testing yet to prove it (I'll do so when I have more of the compute stuff fleshed out) but in theory it should provide similar optimization in Go by reducing the number of pointers and heap objects due to the statically sized arrays. In addition the semantics are different.
   
   ArrayData has an atomic ref count which is managed with a retain and release function. When constructed it automatically calls retain on any buffers and children it was constructed with, etc. ArraySpan doesn't attempt to make any handling of that ref count and isn't considered to own the buffers, and the ArraySpan objects can typically be passed around and copied when needed, adjusting the buffer count for parent buffers only when converted back to an array data at the end.
   
   Thus saving all of the retain and release management during kernel execution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade merged pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
zeroshade merged PR #13929:
URL: https://github.com/apache/arrow/pull/13929


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade commented on a diff in pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
zeroshade commented on code in PR #13929:
URL: https://github.com/apache/arrow/pull/13929#discussion_r951548558


##########
go/arrow/compute/internal/span.go:
##########
@@ -0,0 +1,538 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package internal
+
+import (
+	"reflect"
+	"unsafe"
+
+	"github.com/apache/arrow/go/v10/arrow"
+	"github.com/apache/arrow/go/v10/arrow/array"
+	"github.com/apache/arrow/go/v10/arrow/bitutil"
+	"github.com/apache/arrow/go/v10/arrow/memory"
+	"github.com/apache/arrow/go/v10/arrow/scalar"
+)
+
+// BufferSpan is a lightweight Buffer holder for ArraySpans that does not
+// take ownership of the underlying memory.Buffer at all or could be
+// used to reference raw byte slices instead.
+type BufferSpan struct {
+	// Buf should be the byte slice representing this buffer, if this is
+	// nil then this bufferspan should be considered empty.
+	Buf []byte
+	// Owner should point to an underlying parent memory.Buffer if this
+	// memory is owned by a different, existing, buffer. Retain is not
+	// called on this buffer, so it must not be released as long as
+	// this BufferSpan refers to it.
+	Owner *memory.Buffer
+	// SelfAlloc tracks whether or not this bufferspan is the only owner
+	// of the Owning memory.Buffer. This happens when preallocating
+	// memory or if a kernel allocates it's own buffer for a result.
+	// In these cases, we have to know so we can properly maintain the
+	// refcount if this is later turned into an ArrayData object.
+	SelfAlloc bool
+}
+
+// SetBuffer sets the given buffer into this BufferSpan and marks
+// SelfAlloc as false. This should be called when setting a buffer
+// that is externally owned/created.
+func (b *BufferSpan) SetBuffer(buf *memory.Buffer) {
+	b.Buf = buf.Bytes()
+	b.Owner = buf
+	b.SelfAlloc = false
+}
+
+// WrapBuffer wraps this bufferspan around a buffer and marks
+// SelfAlloc as true. This should be called when setting a buffer
+// that was allocated as part of an execution rather than just
+// re-using an existing buffer from an input array.
+func (b *BufferSpan) WrapBuffer(buf *memory.Buffer) {
+	b.Buf = buf.Bytes()
+	b.Owner = buf
+	b.SelfAlloc = true
+}
+
+// ArraySpan is a light-weight, non-owning version of arrow.ArrayData
+// for more efficient handling with computation and engines. We use
+// explicit go Arrays to define the buffers and some scratch space
+// for easily populating and shifting around pointers to memory without
+// having to worry about and deal with retain/release during calculations.
+type ArraySpan struct {
+	Type    arrow.DataType
+	Len     int64
+	Nulls   int64
+	Offset  int64
+	Buffers [3]BufferSpan
+
+	Scratch [2]uint64

Review Comment:
   yea, it's used for offsets and the union type code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13929:
URL: https://github.com/apache/arrow/pull/13929#issuecomment-1221272604

   @zeroshade Keep in mind that ArraySpan in C++ is merely a performance optimization over ArrayData. That may not make sense or be important for Go.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13929:
URL: https://github.com/apache/arrow/pull/13929#issuecomment-1221272698

   (also, I don't think you need to tag @amol- or @kszucs as reviewers here)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] lidavidm commented on a diff in pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #13929:
URL: https://github.com/apache/arrow/pull/13929#discussion_r951436060


##########
go/arrow/compute/internal/span.go:
##########
@@ -0,0 +1,538 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package internal
+
+import (
+	"reflect"
+	"unsafe"
+
+	"github.com/apache/arrow/go/v10/arrow"
+	"github.com/apache/arrow/go/v10/arrow/array"
+	"github.com/apache/arrow/go/v10/arrow/bitutil"
+	"github.com/apache/arrow/go/v10/arrow/memory"
+	"github.com/apache/arrow/go/v10/arrow/scalar"
+)
+
+// BufferSpan is a lightweight Buffer holder for ArraySpans that does not
+// take ownership of the underlying memory.Buffer at all or could be
+// used to reference raw byte slices instead.
+type BufferSpan struct {
+	// Buf should be the byte slice representing this buffer, if this is
+	// nil then this bufferspan should be considered empty.
+	Buf []byte
+	// Owner should point to an underlying parent memory.Buffer if this
+	// memory is owned by a different, existing, buffer. Retain is not
+	// called on this buffer, so it must not be released as long as
+	// this BufferSpan refers to it.
+	Owner *memory.Buffer
+	// SelfAlloc tracks whether or not this bufferspan is the only owner
+	// of the Owning memory.Buffer. This happens when preallocating
+	// memory or if a kernel allocates it's own buffer for a result.
+	// In these cases, we have to know so we can properly maintain the
+	// refcount if this is later turned into an ArrayData object.
+	SelfAlloc bool
+}
+
+// SetBuffer sets the given buffer into this BufferSpan and marks
+// SelfAlloc as false. This should be called when setting a buffer
+// that is externally owned/created.
+func (b *BufferSpan) SetBuffer(buf *memory.Buffer) {
+	b.Buf = buf.Bytes()
+	b.Owner = buf
+	b.SelfAlloc = false
+}
+
+// WrapBuffer wraps this bufferspan around a buffer and marks
+// SelfAlloc as true. This should be called when setting a buffer
+// that was allocated as part of an execution rather than just
+// re-using an existing buffer from an input array.
+func (b *BufferSpan) WrapBuffer(buf *memory.Buffer) {
+	b.Buf = buf.Bytes()
+	b.Owner = buf
+	b.SelfAlloc = true
+}
+
+// ArraySpan is a light-weight, non-owning version of arrow.ArrayData
+// for more efficient handling with computation and engines. We use
+// explicit go Arrays to define the buffers and some scratch space
+// for easily populating and shifting around pointers to memory without
+// having to worry about and deal with retain/release during calculations.
+type ArraySpan struct {
+	Type    arrow.DataType
+	Len     int64
+	Nulls   int64
+	Offset  int64
+	Buffers [3]BufferSpan
+
+	Scratch [2]uint64

Review Comment:
   (IIRC, the scratch space is for dealing with offsets right? Might be good to document what this field is for since it's not immediately obvious)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] ursabot commented on pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13929:
URL: https://github.com/apache/arrow/pull/13929#issuecomment-1222729912

   Benchmark runs are scheduled for baseline = 8c17925a63fb1d79401c50c82eefb995996b2179 and contender = 258173da59a6b4be663f2d8f6efba017ba3be4f4. 258173da59a6b4be663f2d8f6efba017ba3be4f4 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/4f1d15226ec8450eb13a247ce9eeb01a...07caaae583194bd88eedfd818ebab2c2/)
   [Failed :arrow_down:0.48% :arrow_up:0.24%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/3582a822ab0d4a5aa644b4bad2c9c0ea...a3ae0ed580c44d2c88c64ceae89cf328/)
   [Failed :arrow_down:0.82% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/47f3745b6a0646f1a94e23eb493a2e9c...7cf6d74a562849ffbaaa2a944692b718/)
   [Finished :arrow_down:0.53% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/6ae04333375e464a943eff98b1cc7ee0...5886b8a303a74f08b696f311baf4ae62/)
   Buildkite builds:
   [Finished] [`258173da` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1345)
   [Failed] [`258173da` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1361)
   [Failed] [`258173da` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1343)
   [Finished] [`258173da` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1360)
   [Finished] [`8c17925a` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1344)
   [Failed] [`8c17925a` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1360)
   [Failed] [`8c17925a` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1342)
   [Finished] [`8c17925a` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1359)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13929:
URL: https://github.com/apache/arrow/pull/13929#issuecomment-1221118238

   https://issues.apache.org/jira/browse/ARROW-17479


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] zeroshade commented on pull request #13929: ARROW-17479: [Go] Add ArraySpan and utilities

Posted by GitBox <gi...@apache.org>.
zeroshade commented on PR #13929:
URL: https://github.com/apache/arrow/pull/13929#issuecomment-1221331447

   @lidavidm that's fair point, maybe `ArrayView`? Or `ArraySlice`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org