You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/15 20:45:35 UTC
[GitHub] [arrow] minyoung opened a new pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
minyoung opened a new pull request #12641:
URL: https://github.com/apache/arrow/pull/12641
`arrToWrite` is given to `newMultipathLevelBuilder`, which does call `Retain` on the provided array, so `NewColumnWriter` can `arrToWrite.Release`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
zeroshade commented on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1068653922
@minyoung Looks like your change is causing a "too many releases" in one of the other tests. Make sure to run the tests with the `-tags assert` build tag to reproduce locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1071236053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1071236053
Benchmark runs are scheduled for baseline = 868bc7ee2aeffaded055b46dd72040667f499b11 and contender = 331fc4476b6c38c85d4403f1b9726111b4c28f26. 331fc4476b6c38c85d4403f1b9726111b4c28f26 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34856a07d15e4645a5a5e12b97fc6176...55b94f53ea87449083d6e7b67c8f3ca0/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/1a232dcc4a1748b0834cdfedc02c3ac9...ecc3532adad14c65a763214a52a2ab5a/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/113e53c0eaf24e9998f1854764147ef7...b2eeb4acd21046d49691b95213d855c7/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/de0bde8e848643d19434dc5a5a6dc8dd...48f389de751c4ffaa0f76dba16ccf1db/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade closed pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
zeroshade closed pull request #12641:
URL: https://github.com/apache/arrow/pull/12641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1071236053
Benchmark runs are scheduled for baseline = 868bc7ee2aeffaded055b46dd72040667f499b11 and contender = 331fc4476b6c38c85d4403f1b9726111b4c28f26. 331fc4476b6c38c85d4403f1b9726111b4c28f26 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34856a07d15e4645a5a5e12b97fc6176...55b94f53ea87449083d6e7b67c8f3ca0/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/1a232dcc4a1748b0834cdfedc02c3ac9...ecc3532adad14c65a763214a52a2ab5a/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/113e53c0eaf24e9998f1854764147ef7...b2eeb4acd21046d49691b95213d855c7/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/de0bde8e848643d19434dc5a5a6dc8dd...48f389de751c4ffaa0f76dba16ccf1db/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] minyoung commented on a change in pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
minyoung commented on a change in pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#discussion_r828322776
##########
File path: go/parquet/pqarrow/path_builder.go
##########
@@ -555,7 +555,6 @@ type multipathLevelResult struct {
}
func (m *multipathLevelResult) Release() {
- m.leafArr.Release()
Review comment:
`pathBuilder.addTerminalInfo` itself does call `paths.primitiveArr.Retain` (and a corresponding `Release`), so I think this is safe to remove? This would also explain why the leak only happens with nested types (because they have an extra array not released by this `leafArr.Release`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1071236053
Benchmark runs are scheduled for baseline = 868bc7ee2aeffaded055b46dd72040667f499b11 and contender = 331fc4476b6c38c85d4403f1b9726111b4c28f26. 331fc4476b6c38c85d4403f1b9726111b4c28f26 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34856a07d15e4645a5a5e12b97fc6176...55b94f53ea87449083d6e7b67c8f3ca0/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/1a232dcc4a1748b0834cdfedc02c3ac9...ecc3532adad14c65a763214a52a2ab5a/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/113e53c0eaf24e9998f1854764147ef7...b2eeb4acd21046d49691b95213d855c7/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/de0bde8e848643d19434dc5a5a6dc8dd...48f389de751c4ffaa0f76dba16ccf1db/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1071236053
Benchmark runs are scheduled for baseline = 868bc7ee2aeffaded055b46dd72040667f499b11 and contender = 331fc4476b6c38c85d4403f1b9726111b4c28f26. 331fc4476b6c38c85d4403f1b9726111b4c28f26 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34856a07d15e4645a5a5e12b97fc6176...55b94f53ea87449083d6e7b67c8f3ca0/)
[Finished :arrow_down:0.25% :arrow_up:0.04%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/1a232dcc4a1748b0834cdfedc02c3ac9...ecc3532adad14c65a763214a52a2ab5a/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/113e53c0eaf24e9998f1854764147ef7...b2eeb4acd21046d49691b95213d855c7/)
[Finished :arrow_down:0.13% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/de0bde8e848643d19434dc5a5a6dc8dd...48f389de751c4ffaa0f76dba16ccf1db/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] minyoung commented on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
minyoung commented on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1069401882
@zeroshade thanks for the `-tags assert`! Digging into the "too many releases" now, seems like this is more subtle than I originally hoped...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1068454113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1071236053
Benchmark runs are scheduled for baseline = 868bc7ee2aeffaded055b46dd72040667f499b11 and contender = 331fc4476b6c38c85d4403f1b9726111b4c28f26. 331fc4476b6c38c85d4403f1b9726111b4c28f26 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34856a07d15e4645a5a5e12b97fc6176...55b94f53ea87449083d6e7b67c8f3ca0/)
[Finished :arrow_down:0.25% :arrow_up:0.04%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/1a232dcc4a1748b0834cdfedc02c3ac9...ecc3532adad14c65a763214a52a2ab5a/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/113e53c0eaf24e9998f1854764147ef7...b2eeb4acd21046d49691b95213d855c7/)
[Finished :arrow_down:0.13% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/de0bde8e848643d19434dc5a5a6dc8dd...48f389de751c4ffaa0f76dba16ccf1db/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12641:
URL: https://github.com/apache/arrow/pull/12641#issuecomment-1071236053
Benchmark runs are scheduled for baseline = 868bc7ee2aeffaded055b46dd72040667f499b11 and contender = 331fc4476b6c38c85d4403f1b9726111b4c28f26. 331fc4476b6c38c85d4403f1b9726111b4c28f26 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34856a07d15e4645a5a5e12b97fc6176...55b94f53ea87449083d6e7b67c8f3ca0/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/1a232dcc4a1748b0834cdfedc02c3ac9...ecc3532adad14c65a763214a52a2ab5a/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/113e53c0eaf24e9998f1854764147ef7...b2eeb4acd21046d49691b95213d855c7/)
[Finished :arrow_down:0.13% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/de0bde8e848643d19434dc5a5a6dc8dd...48f389de751c4ffaa0f76dba16ccf1db/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade closed pull request #12641: ARROW-15946: [Go] Fix memory leak in pqarrow.NewColumnWriter when writing nested data
Posted by GitBox <gi...@apache.org>.
zeroshade closed pull request #12641:
URL: https://github.com/apache/arrow/pull/12641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org