You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pixelherodev (via GitHub)" <gi...@apache.org> on 2025/10/30 22:09:53 UTC

[PR] flatbuf: regenerate using optimized flatc [arrow-go]

pixelherodev opened a new pull request, #559:
URL: https://github.com/apache/arrow-go/pull/559

   Regenerates the flatbuf package using the optimized flatbuffer compiler [here](https://github.com/pixelherodev/flatbuffers/tree/opt). It's a pretty straightforward patch; it basically just passes internal values along the stack instead of by pointers (which necessitates heaps allocations).
   
   I know editing generated code is usually not looked on super favorably, but IMO the performance win is sufficient to justify it.
   
   Also addresses the `// FIXME(sbinet) refactor msg-header handling.`; with the Table exposed directly (which it should be since it can be modified via .Init anyways!), we can just use e.g. msg.Header(&foo.Table) instead of needing the initFB helper.
   
   Before:
   
   ```
   BenchmarkIPC/Writer/codec=plain-24         	 223521	     5303 ns/op	   8160 B/op	     91 allocs/op
   BenchmarkIPC/Reader/codec=plain-24         	 223260	     4752 ns/op	   6130 B/op	     83 allocs/op
   BenchmarkIPC/Writer/codec=zstd-24          	   5019	   300874 ns/op	2338732 B/op	    150 allocs/op
   BenchmarkIPC/Reader/codec=zstd-24          	  37974	    28207 ns/op	  25877 B/op	    194 allocs/op
   BenchmarkIPC/Writer/codec=lz4-24           	  96709	    11894 ns/op	  13021 B/op	    134 allocs/op
   BenchmarkIPC/Reader/codec=lz4-24           	 120859	     9945 ns/op	  10155 B/op	    124 allocs/op
   ```
   
   After:
   
   ```
   cpu: AMD Ryzen 9 7900X3D 12-Core Processor          
   BenchmarkIPC/Writer/codec=plain-24         	 213406	     5856 ns/op	   8160 B/op	     91 allocs/op
   BenchmarkIPC/Reader/codec=plain-24         	 270422	     4526 ns/op	   5780 B/op	     69 allocs/op
   BenchmarkIPC/Writer/codec=zstd-24          	   5756	   335170 ns/op	2338710 B/op	    150 allocs/op
   BenchmarkIPC/Reader/codec=zstd-24          	  42300	    25984 ns/op	  25470 B/op	    178 allocs/op
   BenchmarkIPC/Writer/codec=lz4-24           	 112836	    10816 ns/op	  12701 B/op	    134 allocs/op
   BenchmarkIPC/Reader/codec=lz4-24           	 140852	     9199 ns/op	   9677 B/op	    108 allocs/op
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3482499044

   Hmm, I'd be fine with creating a repo for the patches myself. @lidavidm @kou @amoeba what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3475762338

   > Is there any possibility of getting the optimizations pushed upstream in the flatbuffer compiler?
   
   I'm trying, but realistically it's a Google project. I honestly doubt they're going to even look at it; they get bonuses for ruining public-facing projects, not for making things better.
   
   > Otherwise we should at least update the README or something to ensure that future regenerating of the flatbuffer files uses the same compiler patches
   
   That's definitely fair - would you want to host the compiler patches somewhere yourself? It's one reason I wanted to split this out into its own PR; I'm not sure the best way to handle it. If it was my project, I'd just link to my own patches, but - it's not :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3578208452

   This is a performance optimization. Some benchmark results (like you showed in this PR's description) are also helpful for reviewers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3720967903

   @pixelherodev @kou where do we stand on this so far?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3560571637

   So, what's the plan here? I think google's a lost cause...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3484079864

   > Do we have a sense of how hard it would be to keep these patches maintained?
   
   Can't say for sure, given that I don't know what upstream looks like, but FWIW: I developed the patches on internal/flatbuf first, not knowing it was generated code, and then it took me all of... ten minutes? to figure out how to patch the Go code generator, and that's without having touched C++ in ~5 years.
   
   So, I'd guess "not very hard." Especially since the logic of the patch is quite straightforwards:
   
   - Inline the flatbuffer Table. This means altering the `_tab flatbuffer.Table` line, which is constant, to just drop the _tab.
       - Find all references to _tab and replace them with direct references. Was able to just s/_tab\.//g and it worked right away.
   - Find the function that generates a pointer return and takes a pointer in to use as storage, rewrite the return signature from *T to `(T, bool)`, drop the logic that checks the passed in pointer and allocates, and just return the structure on success.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3484996106

   Rebased and lint fixed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3484020630

   Ah, sorry. You already mentioned the CLA status.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3676676218

   > @pixelherodev Could you update the description of [google/flatbuffers#8744](https://github.com/google/flatbuffers/pull/8744) with more details for reviewers like you did in [#559 (comment)](https://github.com/apache/arrow-go/pull/559#issuecomment-3484079864) ?
   > 
   > Then, let's ping to a FlatBuffers Go maintainer ( https://github.com/dbaileychess ?) before we proceed this.
   
   Yep, will do!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3484018729

   Thanks. It seems that your need to sign a CLA: https://github.com/google/flatbuffers/pull/8744#issuecomment-3470448944
   
   Could you check it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3578203456

   @pixelherodev Could you update the description of https://github.com/google/flatbuffers/pull/8744 with more details for reviewers like you did in https://github.com/apache/arrow-go/pull/559#issuecomment-3484079864 ?
   
   Then, let's ping to a FlatBuffers Go maintainer ( https://github.com/dbaileychess ?) before we proceed this.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3483512032

   > (Also, is there a PR upstream that we can look at? I agree that Google seems unlikely to care, but we should at least try.)
   
   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3483455705

   (Also, is there a PR upstream that we can look at? I agree that Google seems unlikely to care, but we should at least try.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3576319974

   @lidavidm @kou @amoeba I'm open to creating a personal repo with the patches for maintaining, or we can explicitly create an arrow-prefixed repo in the apache org if you think that would be more appropriate here.
   
   Looking for feedback rather than me unilaterally deciding to go for it, thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3955006235

   Left work shortly after you asked, and house caught fire not too long after that.
   
   I'll try to get around to this soon, but it's not a priority for me right now :upside_down_face: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3483454429

   Do we have a sense of how hard it would be to keep these patches maintained?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3483874365

   https://github.com/google/flatbuffers/pull/8744 <- yes, I did open one upstream @lidavidm @kou. If they engage with me I intend to push for it.
   
   I need to figure out the CLA status and such though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3484070538

   Yeah, I need to double check. Contributing to Arrow is definitely fine, but Google's asking for more than just a contribution under an Apache license, they effectively want total ownership in all but name, and I don't think I _want_ to try and get permission to give them that, frankly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3483953647

   Thanks! I'm going to try poking around to see if I can find any chain that leads to a Flatbuffer owner but, well, I have low hopes...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "pixelherodev (via GitHub)" <gi...@apache.org>.
pixelherodev commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3484093848

   > not very hard
   
   FWIW: simple enough that I'd be totally fine committing to maintaining it myself, if we knew each other enough for that to mean anything :sweat_smile: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3473355197

   Is there any possibility of getting the optimizations pushed upstream in the flatbuffer compiler?
   
   Otherwise we should at least update the README or something to ensure that future regenerating of the flatbuffer files uses the same compiler patches


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] flatbuf: regenerate using optimized flatc [arrow-go]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #559:
URL: https://github.com/apache/arrow-go/pull/559#issuecomment-3578076499

   I'm not sure we can add the code to an Apache-controlled repo without going through some hoops. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org