You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Douglas Creager (JIRA)" <ji...@apache.org> on 2011/01/24 22:23:44 UTC
[jira] Created: (AVRO-746) Atomic reference counts
Atomic reference counts
-----------------------
Key: AVRO-746
URL: https://issues.apache.org/jira/browse/AVRO-746
Project: Avro
Issue Type: Bug
Components: c
Reporter: Douglas Creager
Assignee: Douglas Creager
Attachments: 0001-Atomic-reference-counts.patch
In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts are updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: 0001-Atomic-reference-counts.patch
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: (was: fixed-make-dist.patch)
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: 0001-Atomic-reference-counts.patch
Here's a new version of the patch that doesn't include the full OpenPA distribution. There's a single header file, src/refcount.h, which defines a bunch of static inline functions.
I think this patch takes care of all of the platforms that Bruce mentioned. There are implementations using Mac OS X functions (which should also include iOS), GCC intrinsics (valid for any platform with GCC >= 4.1), raw assembly in GCC for i386, x86_64, and ppc, and Windows intrinsics. I don't have a Windows machine to test the Windows intrinsics on; Bruce, would you be able to test that?
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-746) Atomic reference counts
Posted by "Bruce Mitchener (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988832#comment-12988832 ]
Bruce Mitchener commented on AVRO-746:
--------------------------------------
I'd like to see more / wider discussion of this.
1) Portability: Will this work on Windows as set up now via CMake?
2) New external code...
3) Performance hit in single-threaded case?
4) Portability: iOS / Android impact?
Also, the CMake integration here doesn't build any code, it just uses the headers, I'm guessing that's okay?
And this piles on top of some other stuff that uses C99 struct initializers ... I have another patch that I've been working on that gets rid of those because they don't work in Visual Studio which will now conflict with this.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, fixed-make-dist.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: 0001-Performance-test-program.patch
Here's a patch that does a performance test of the reference counting mechanism. It allocates a single int value, and then references and dereferences that value 100,000,000 times. It runs the test 3 times and prints out the average time across the 3 runs.
It turns out that the atomic counts are a bigger performance hit than I thought. With the original implementation, which just uses C's ++ and -- operators, the test case ran in 0.460 seconds. Using the new atomic operation, it took 1.820 seconds.
That said, I don't know if reference counting is the bottleneck in a "typical" use of the library. I'm going to add some more performance tests to the test program to see what we get.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, 0001-Performance-test-program.patch, fixed-make-dist.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064156#comment-13064156 ]
Douglas Creager commented on AVRO-746:
--------------------------------------
Apologies! CHANGES.txt now updated.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Fix For: 1.6.0
>
> Attachments: 0001-Atomic-reference-counts.patch, 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: (was: fixed-make-dist.patch)
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, fixed-make-dist.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989897#comment-12989897 ]
Douglas Creager commented on AVRO-746:
--------------------------------------
I'm not married to the idea of incorporating OpenPA, it just seemed like an expedient solution. We could easily create our own refcount.h header, that contained static inline definitions like OpenPA's does, to ensure that we can target all of the platforms you mention.
I'll try to run some tests to compare single-threaded performance. I don't think it's a big hit, but some data to back that up would be nice.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, fixed-make-dist.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AVRO-746) Atomic reference counts
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993753#comment-12993753 ]
Scott Carey commented on AVRO-746:
----------------------------------
That sounds about right. Atomic increments are a lot more expensive than a simple increment, since it has to push the value to the processor cache and notify any other processors that may have copies of the cache line. And because an ordinary increment is basically the cheapest thing you can ask a CPU to do. However, such increments are rarely a big chunk of the total work.
Very heavy multi-threaded access will likely show a bigger performance hit.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch, fixed-make-dist.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: fixed-make-dist.patch
Ignore the 0001 patch; the new openpa directory wasn't being included in the distribution tarball, giving you compilation errors. The new patch fixes this.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, fixed-make-dist.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: 0001-Atomic-reference-counts.patch
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts are updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: 0002-Serialization-performance-test.patch
Here's another performance test. It basically reproduces the write/read test from test_avro_data, but does it 100,000 times and times it.
In this case, the reference counting isn't as much of a bottleneck. I'm averaging about 0.85 seconds per run with the original non-atomic reference counts, and about 0.87 seconds with atomic reference counts.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch, fixed-make-dist.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: fixed-make-dist.patch
Third time's a charm.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, fixed-make-dist.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: (was: 0001-Atomic-reference-counts.patch)
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager resolved AVRO-746.
----------------------------------
Resolution: Fixed
Fix Version/s: 1.6.0
Committed to SVN for 1.6.0; not back-ported to 1.5 branch
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Fix For: 1.6.0
>
> Attachments: 0001-Atomic-reference-counts.patch, 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-746) Atomic reference counts
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064141#comment-13064141 ]
Doug Cutting commented on AVRO-746:
-----------------------------------
Can you please update the top-level CHANGES.txt too? Thanks!
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Fix For: 1.6.0
>
> Attachments: 0001-Atomic-reference-counts.patch, 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Description:
In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
was:
In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts are updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045507#comment-13045507 ]
Douglas Creager commented on AVRO-746:
--------------------------------------
There hasn't been any word on whether the Windows intrinsics work, but this is working on all of the other platforms I can test. And we still don't officially list Windows as a supported port; AVRO-551 is still open to get the tests to compile and run under Windows. Unless I hear objections in the next couple of days, I'm going to go ahead and commit this on the 1.6 trunk branch.
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch, 0001-Performance-test-program.patch, 0002-Serialization-performance-test.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AVRO-746) Atomic reference counts
Posted by "Douglas Creager (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Douglas Creager updated AVRO-746:
---------------------------------
Attachment: (was: 0001-Atomic-reference-counts.patch)
> Atomic reference counts
> -----------------------
>
> Key: AVRO-746
> URL: https://issues.apache.org/jira/browse/AVRO-746
> Project: Avro
> Issue Type: Bug
> Components: c
> Reporter: Douglas Creager
> Assignee: Douglas Creager
> Attachments: 0001-Atomic-reference-counts.patch
>
>
> In one of my projects that uses Avro, I pass avro_datum_t instances between threads, using the reference count mechanism to make sure that they're not freed while any thread still has a reference to them. I was getting some spurious segfaults, which were caused by the fact that the reference counts aren't updated atomically. I've created a patch that implements atomic reference counts, using the [OpenPA library|http://trac.mcs.anl.gov/projects/openpa/] to provide the atomic operations themselves. (That library is MIT licensed, so it can be included in the source tree.)
> Note that only avro_XXX_incref and avro_XXX_decref are thread-safe as a result of this patch. For all of the other library functions, the caller is still responsible for ensuring thread safety.
> The patch makes sure that the OpenPA code works in both the old autotools build and the newer CMake build.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.