You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pixelherodev (via GitHub)" <gi...@apache.org> on 2025/11/10 21:07:50 UTC

[PR] String optimize [arrow-go]

pixelherodev opened a new pull request, #570:
URL: https://github.com/apache/arrow-go/pull/570

   Slight speed-up for a few string conversions.
   
   Strings are immutable. As such, `string([]byte(foo))`, given a mutable buffer, must copy the string, so that changes to the underlying slice do not break the language's requirements.
   
   `unsafe.String(foo, length)`, instead, crafts a string header pointing to the existing buffer. _So long as that buffer is never modified_ for the lifetime of the string, this can be totally safe, and is faster.
   
   Since the flatbufs will never be modified, it's totally safe to craft strings that point directly into its innards. The string pointer is sufficient to prevent GC, and we never modify that flatbuf data on read paths.
   
   Tests pass and there's a small but measurable performance improvement in profiles I was looking at.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] String optimize [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #570:
URL: https://github.com/apache/arrow-go/pull/570#issuecomment-3513905726

   do we see any measurable performance difference in the existing benchmarks?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] String optimize [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #570:
URL: https://github.com/apache/arrow-go/pull/570#issuecomment-3560192874

   I don't think so.
   
   For the profiles I was looking at, the rate of timestamps was sufficient that this minor savings was happening millions of times per second, and was _still_ pretty small, I think. I can double check, but this one might be worth skipping, yeah.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] String optimize [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #570:
URL: https://github.com/apache/arrow-go/pull/570#issuecomment-3683463162

   @pixelherodev can you fix the lint issue any the failures here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] String optimize [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #570:
URL: https://github.com/apache/arrow-go/pull/570#discussion_r3876193537


##########
arrow/ipc/metadata.go:
##########
@@ -183,7 +185,8 @@ func fieldFromFB(field *flatbuf.Field, pos dictutils.FieldPos, memo *dictutils.M
 		o   arrow.Field
 	)
 
-	o.Name = string(field.Name())
+	name := field.Name()
+	o.Name = unsafe.String(&name[0], len(name))

Review Comment:
   **Blocking:** Arrow permits empty field names, so `&name[0]` panics when `len(name) == 0`. More broadly, `unsafe.String` makes the decoded schema borrow FlatBuffer storage; with `NewMappedFileReader`, that storage can be caller-owned or memory-mapped, and mutating it changes the already-decoded field name. Please retain the owned conversion:
   
   ```go
   o.Name = string(field.Name())
   ```
   
   `unsafe.SliceData` would only address empty input, not ownership.



##########
arrow/ipc/metadata.go:
##########
@@ -986,8 +989,9 @@ func timeFromFB(data flatbuf.Time) (arrow.DataType, error) {
 
 func timestampFromFB(data flatbuf.Timestamp) (arrow.DataType, error) {
 	unit := unitFromFB(data.Unit())
-	tz := string(data.Timezone())
-	return &arrow.TimestampType{Unit: unit, TimeZone: tz}, nil
+	tz := data.Timezone()
+	tzs := unsafe.String(&tz[0], len(tz))

Review Comment:
   **Blocking:** Unzoned Arrow timestamps have an empty timezone, so this panics on `&tz[0]`. The resulting string also borrows FlatBuffer storage. Please retain an owned conversion:
   
   ```go
   tz := string(data.Timezone())
   return &arrow.TimestampType{Unit: unit, TimeZone: tz}, nil
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] String optimize [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #570:
URL: https://github.com/apache/arrow-go/pull/570#issuecomment-5243129343

   @pixelherodev A few things need addressing before review — see our [Pull Request quality criteria](https://github.com/apache/arrow-go/blob/main/CONTRIBUTING.md).
   
   - :x: **Lint / pre-commit** — failing: `Lint`. See [docs](https://github.com/apache/arrow-go/blob/main/CONTRIBUTING.md).
   - :x: **Platform tests** — failing: `AMD64 Debian 12 Go 1.24`, `ARM64 Debian 12 Go 1.24`, `Verify (macos-latest)`, `AMD64 Debian 12 Go 1.24 - CGO`, `Verify (ubuntu-latest)` (+5 more). See [docs](https://arrow.apache.org/docs/developers/index.html).
   
   > **Note:** Your branch is **432 commits behind `main`**. Please rebase and push again to get up-to-date CI results.
   
   No rush.
   
   ---
   
   _Note: This comment was drafted by an AI-assisted triage tool run by a maintainer, and may contain mistakes. Once you have addressed the points above, an Apache Arrow Go maintainer — a real person — will take the next look at your PR. If anything here looks wrong, say so on the PR and a maintainer will follow up. See [CONTRIBUTING.md](https://github.com/apache/arrow-go/blob/main/CONTRIBUTING.md) for the project's contribution conventions._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org