You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "quinnj (via GitHub)" <gi...@apache.org> on 2023/05/25 22:10:22 UTC
[GitHub] [arrow-julia] quinnj opened a new pull request, #445: Refactor compressors/decompressors for laziness + safety
quinnj opened a new pull request, #445:
URL: https://github.com/apache/arrow-julia/pull/445
Fixes #396.
As noted in the originally reported issue, enabling debug logging when writing arrow data with compression can result in segfaults because the underlying CodecX package have debug logs, causing task switches/migration and thus making the pattern of using a single `X_COMPRESSOR` array indexed by `Threads.threadid()` unsafe since multiple threads may try using the same compressor at the same time.
We fix this by wrapping each compressor in a `Lockable` and ensuring the `compress` (or `uncompress`) operation holds the lock for the duration of the operation. We also:
* Add a decompressor per thread to avoid recreating them over and over during reading
* Lazily initialize compressors/decompressors in a way that is 1.9+ safe and only creates the object when needed by a specific thread
* Switch from WorkerUtilities -> ConcurrentUtilities (the package was renamed)
Successor to https://github.com/apache/arrow-julia/pull/397; I've added @svilupp as a co-author here since they started the original movement for the code to go in this direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] quinnj merged pull request #445: Refactor compressors/decompressors for laziness + safety
Posted by "quinnj (via GitHub)" <gi...@apache.org>.
quinnj merged PR #445:
URL: https://github.com/apache/arrow-julia/pull/445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] codecov-commenter commented on pull request #445: Refactor compressors/decompressors for laziness + safety
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #445:
URL: https://github.com/apache/arrow-julia/pull/445#issuecomment-1563575108
## [Codecov](https://app.codecov.io/gh/apache/arrow-julia/pull/445?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#445](https://app.codecov.io/gh/apache/arrow-julia/pull/445?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (bed76e0) into [main](https://app.codecov.io/gh/apache/arrow-julia/commit/94f7e37f637878c9850ebe345664011a636e3ecb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (94f7e37) will **decrease** coverage by `82.36%`.
> The diff coverage is `0.00%`.
> :exclamation: Current head bed76e0 differs from pull request most recent head d0f9a76. Consider uploading reports for the commit d0f9a76 to get more accurate results
```diff
@@ Coverage Diff @@
## main #445 +/- ##
==========================================
- Coverage 87.25% 4.90% -82.36%
==========================================
Files 26 25 -1
Lines 3210 3182 -28
==========================================
- Hits 2801 156 -2645
- Misses 409 3026 +2617
```
| [Impacted Files](https://app.codecov.io/gh/apache/arrow-julia/pull/445?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [src/Arrow.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/445?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL0Fycm93Lmps) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [src/arraytypes/arraytypes.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/445?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL2FycmF5dHlwZXMvYXJyYXl0eXBlcy5qbA==) | `0.00% <0.00%> (-89.63%)` | :arrow_down: |
| [src/table.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/445?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL3RhYmxlLmps) | `0.00% <0.00%> (-92.91%)` | :arrow_down: |
| [src/write.jl](https://app.codecov.io/gh/apache/arrow-julia/pull/445?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL3dyaXRlLmps) | `0.00% <0.00%> (-97.60%)` | :arrow_down: |
... and [20 files with indirect coverage changes](https://app.codecov.io/gh/apache/arrow-julia/pull/445/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org