You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stdcxx.apache.org by Martin Sebor <ms...@gmail.com> on 2012/10/01 00:18:57 UTC

Re: Fwd: Re: STDCXX-1071 numpunct facet defect

I see you did a 64-bit build while I did a 32-bit one. so
I tried 64-bits. The cached version (i.e., the one compiled
with -UNO_USE_NUMPUNCT_CACHE) is still about twice as fast
as the non-cached one (compiled with -DNO_USE_NUMPUNCT_CACHE).

I had made one change to the test program that I thought might
account for the difference: I removed the call to abort from
the thread function since it was causing the process to exit
prematurely in some of my tests. But since you used the
modified program for your latest measurements that couldn't
be it.

I can't explain the differences. They just don't make sense
to me. Your results should be the other way around. Can you
post the disassembly of function f() for each of the two
configurations of the test?

Martin

On 09/30/2012 03:30 PM, Liviu Nicoara wrote:
> On 9/30/12 2:21 PM, Liviu Nicoara wrote:
>> Forwarding with the attachment.
>>
>> -------- Original Message --------
>> Subject: Re: STDCXX-1071 numpunct facet defect
>> Date: Sun, 30 Sep 2012 12:09:10 -0600
>> From: Martin Sebor <ms...@gmail.com>
>> To: Liviu Nicoara <ni...@hates.ms>
>>
>>> On 9/27/12 8:27 PM, Martin Sebor wrote:
>>
>> Here are my timings for library-reduction.cpp when compiled
>> GCC 4.5.3 on Solaris 10 (4 SPARCV9 CPUs). I had to make a small
>> number of trivial changes to get it to compile:
>>
>> With cache No cache
>> real 1m38.332s 8m58.568s
>> user 6m30.244s 34m25.942s
>> sys 0m0.060s 0m3.922s
>>
>> I also experimented with the program on Linux (CEL 4 with 16
>> CPUs). Initially, I saw no differences between the two versions.
>> So I modified it a bit to make it closer to the library (the
>> modified program is attached). With those changes the timings
>
> I see the difference -- your program has a virtual function it calls
> from the inline grouping function.
>
>> are below:
>>
>> With cache No cache
>> real 0m 1.107s 0m 5.669s
>> user 0m17.204s 0m 5.669s
>> sys 0m 0.000s 0m22.347s
>>
>> I also recompiled and re-ran the test on Solaris. To speed
>> things along, I set the number threads and loops to 8 and
>> 1000000. The numbers are as follows:
>>
>> With cache No cache
>> real 0m3.341s 0m26.333s
>> user 0m13.052s 1m37.470s
>> sys 0m0.009s 0m0.132s
>>
>> The numbers match my expectation. The overhead without the
>> "numpunct cache" is considerable.
>
> I have done another (smaller) round of measurements, this time using the
> test program you posted. Here are the results:
>
> * iMac, 4x Intel, 12S:
>
> 16, 10000000:
>
> Cached Not cached
> real 0m9.300s 0m5.224s
> user 0m36.441s 0m20.523s
> sys 0m0.043s 0m0.068s
>
> * iMac, 4x Intel, 12D:
>
> Cached Not cached
> real 0m9.012s 0m5.774s
> user 0m35.343s 0m20.997s
> sys 0m0.045s 0m0.183s
>
> * Linux Slackware, 16x AMD Opteron, 12S:
>
> 16, 10000000:
>
> Cached Not cached
> real 0m29.798s 0m3.278s
> user 0m48.662s 0m47.338s
> sys 6m18.525s 0m3.298s
>
>>
>> Somewhat unexpectedly, the test with the cache didn't crash.
>
> On my iMac it did not crash for me either (gcc 4.5.4), this time. On the
> other box (gcc 4.5.2) crashed every time with caching, so I had to add a
> call to fac.grouping outside the thread function to initialize the "facet".
>
> Liviu


Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Liviu Nicoara <ni...@hates.ms>.
On 10/03/12 11:10, Martin Sebor wrote:
> [...]
> I was just thinking of a few simple loops along the lines of:
>

tl;dr: I consider the results of the multi-threaded performance tests (12S, Intel/AMD multicores) as coming from heavy contention in copying of reference-counted std::string objects.

The long version: I went back to the original test case (attached, t.cpp). As a reminder, the program is a simplification of the library numpunct facet creation process, which can be compiled either with data caching enabled or disabled.

I ran the perf tool (https://perf.wiki.kernel.org/index.php/Main_Page) on the program, and I noticed marked differences in the number of instructions and cycles (an order of magnitude) and also in the number of CPU-migrations and context-switches (way larger for the cached version):

  Performance counter stats for './t':

       121168.026301  task-clock-msecs         #      3.778 CPUs
             6133377  context-switches         #      0.051 M/sec
               35457  CPU-migrations           #      0.000 M/sec
                 298  page-faults              #      0.000 M/sec
        338006437277  cycles                   #   2789.568 M/sec
        136554855096  instructions             #      0.404 IPC
         31298346444  branches                 #    258.305 M/sec
          2305761092  branch-misses            #      7.367 %
          1592157186  cache-references         #     13.140 M/sec
             1729755  cache-misses             #      0.014 M/sec

        32.070381363  seconds time elapsed

vs.

  Performance counter stats for './t':

        16194.235282  task-clock-msecs         #      3.883 CPUs
                7491  context-switches         #      0.000 M/sec
                  47  CPU-migrations           #      0.000 M/sec
                 311  page-faults              #      0.000 M/sec
         45210281987  cycles                   #   2791.752 M/sec
         76784904688  instructions             #      1.698 IPC
         18891711522  branches                 #   1166.570 M/sec
          3162251552  branch-misses            #     16.739 %
             3332839  cache-references         #      0.206 M/sec
              119424  cache-misses             #      0.007 M/sec

         4.170904279  seconds time elapsed


Then, I have run more detailed reports on the runs. I noticed that the report for the cached version contains what appears to be thread synchronization overhead, whereas the non-cached version doesn't:

# Events: 134K cycles
#
# Overhead  Command       Shared Object                               Symbol
# ........  .......  ..................  ...................................
#
     50.69%        t  [kernel.kallsyms]   [k] _raw_spin_lock
      5.46%        t  libpthread-2.13.so  [.] pthread_mutex_lock
      4.16%        t  libpthread-2.13.so  [.] __pthread_mutex_unlock
      3.65%        t  libpthread-2.13.so  [.] __lll_lock_wait
      3.35%        t  [kernel.kallsyms]   [k] copy_user_generic_string
      3.09%        t              40395b  [.]           40395b
      2.91%        t  [kernel.kallsyms]   [k] hash_futex
      2.87%        t  [kernel.kallsyms]   [k] futex_wake
      2.67%        t  t                   [.] f
      2.37%        t  [kernel.kallsyms]   [k] futex_wait_setup
      1.61%        t  [kernel.kallsyms]   [k] system_call
      1.00%        t  [kernel.kallsyms]   [k] system_call_after_swapgs
      0.98%        t  [kernel.kallsyms]   [k] __switch_to
      0.96%        t  libpthread-2.13.so  [.] __lll_unlock_wake
      0.95%        t  [kernel.kallsyms]   [k] get_futex_key
      0.92%        t  libc-2.13.so        [.] __strlen_sse42
      [...]


vs.


# Events: 15K cycles
#
# Overhead  Command       Shared Object                                                                                                                          Symbol
# ........  .......  ..................  ..............................................................................................................................
#
     49.55%   :29571        7f1ed3830a86  [.]     7f1ed3830a86
      8.67%        t  libc-2.13.so        [.] __malloc
      8.03%        t  libc-2.13.so        [.] cfree
      4.88%        t  t                   [.] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)
      4.67%        t  libc-2.13.so        [.] __strlen_sse42
      4.14%        t  libpthread-2.13.so  [.] __pthread_mutex_unlock
      3.62%        t  libpthread-2.13.so  [.] pthread_mutex_lock
      2.64%        t  t                   [.] f
      2.40%        t  libc-2.13.so        [.] _int_malloc
      2.10%        t  libc-2.13.so        [.] _int_free
      1.53%        t  libpthread-2.13.so  [.] __pthread_mutex_init_internal
      1.33%        t  t                   [.] operator new(unsigned long)
      1.08%        t  t                   [.] X::numpunct::do_grouping() const
      1.04%        t  libc-2.13.so        [.] __memcpy_sse2
      0.89%        t  t                   [.] X::get_data(X::facet*)
      0.51%        t  t                   [.] __rw::__rw_allocate(unsigned long, int)
      0.45%        t  t                   [.] strlen@plt
      0.33%        t  t                   [.] __rw::__rw_deallocate(void*, unsigned long, int)
      0.32%        t  libpthread-2.13.so  [.] pthread_mutex_destroy
      0.21%        t  t                   [.] pthread_mutex_unlock@plt
      0.21%        t  t                   [.] pthread_mutex_init@plt
      0.18%        t  t                   [.] pthread_mutex_destroy@plt
      0.17%        t  t                   [.] operator delete(void*)
      0.17%        t  t                   [.] malloc@plt
      0.16%        t  t                   [.] memcpy@plt
      0.15%   :29571  [kernel.kallsyms]   [k] hpet_msi_next_event
      0.12%        t  libc-2.13.so        [.] __strrchr_sse2
      0.12%        t  t                   [.] pthread_mutex_lock@plt
      0.11%        t  t                   [.] free@plt
      [...]


Now, if you remember from previous measurements, the performance of the two versions of the program was reversed for the case of one thread, i.e., one-threaded programs performed better in the cached version. Here is the report for a run with one thread, the cached version (running one thread in 1000M loops, otherwise it does not register anything):

# Events: 36K cycles
#
# Overhead  Command       Shared Object                    Symbol
# ........  .......  ..................  ........................
#
     37.68%        t  libpthread-2.13.so  [.] __pthread_mutex_unlock
     31.04%        t  libpthread-2.13.so  [.] pthread_mutex_lock
     11.87%        t  t                   [.] f
      9.64%   :27362        7fd94e2ebd6d  [.]     7fd94e2ebd6d
      3.21%        t  libc-2.13.so        [.] __strlen_sse42
      2.38%        t  t                   [.] pthread_mutex_unlock@plt
      2.27%        t  t                   [.] pthread_mutex_lock@plt
      1.61%        t  t                   [.] strlen@plt
      [...]

The one-threaded run for the non-cached version yields a report identical with the multi-threaded run.


There are two things which I take from this round of measurements: first, that performance suffers on multicores in heavy contention, possibly as a direct result of std::string object copying and I'll follow up with direct measurements. Second, that perhaps we should not focus on this particular use model for the numpunct facet, because it may not be very representative for `normal' usage.


HTH

Liviu

Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Liviu Nicoara <ni...@hates.ms>.
On 10/04/12 22:41, Liviu Nicoara wrote:
> On 10/4/12 10:10 PM, Liviu Nicoara wrote:
>>
>> On 10/3/12 11:10 AM, Martin Sebor wrote:
>>> On 10/03/2012 07:01 AM, Liviu Nicoara wrote:
>>>>
>>>    void* thread_func (void*) {
>>>        for (int i = 0; i < N; ++)
>>>            test 1: do some simple stuff inline
>>>            test 2: call a virtual function to do the same stuff
>>>            test 3: lock and unlock a mutex and do the same stuff
>>>    }
>>>
>>> Test 1 should be the fastest and test 3 the slowest. This should
>>> hold regardless of what "simple stuff" is (eventually, even when
>>> it's getting numpunct::grouping() data).
>>
>> That is expected; I attached test case x.cpp and results-x.txt.
>>
>> I did not find it too interesting in its own, though. The difference
>> between the
>> cached and non-cached data is that in the case of the cached data the
>> copying of
>> the string involves nothing more than a bump in the reference counter,
>> whereas
>> in the non-cached version a string object is constructed anew, and
>> memory gets
>> allocated for its body. Yet, in my measurements, the cached version is
>> the one
>> which shows the worse performance.
>>
>> So, I extracted the std::string class and simplified it down and put
>> it in
>> another test case. That would be u.cpp and the results are
>> results-u.txt. The
>> results show the same performance trends although the absolute values
>> have
>> skewed. Will get back to this after I digest the results a bit more.

This discussion has stagnated lately. I hope to allocate some time this 
week-end to go over the results, re-verify them and analyze the 
disassembly. In the meantime, any thoughts?

Liviu


Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Liviu Nicoara <ni...@hates.ms>.
On 10/4/12 10:10 PM, Liviu Nicoara wrote:
>
> On 10/3/12 11:10 AM, Martin Sebor wrote:
>> On 10/03/2012 07:01 AM, Liviu Nicoara wrote:
>>>
>>> I am gathering some more measurements along these lines but it's time
>>> consuming. I estimate I will have some ready for review later today or
>>> tomorrow. In the meantime could you please post your kernel, glibc and
>>> compiler versions?
>>
>> I was just thinking of a few simple loops along the lines of:
>>
>>    void* thread_func (void*) {
>>        for (int i = 0; i < N; ++)
>>            test 1: do some simple stuff inline
>>            test 2: call a virtual function to do the same stuff
>>            test 3: lock and unlock a mutex and do the same stuff
>>    }
>>
>> Test 1 should be the fastest and test 3 the slowest. This should
>> hold regardless of what "simple stuff" is (eventually, even when
>> it's getting numpunct::grouping() data).
>
> That is expected; I attached test case x.cpp and results-x.txt.
>
> I did not find it too interesting in its own, though. The difference between the
> cached and non-cached data is that in the case of the cached data the copying of
> the string involves nothing more than a bump in the reference counter, whereas
> in the non-cached version a string object is constructed anew, and memory gets
> allocated for its body. Yet, in my measurements, the cached version is the one
> which shows the worse performance.
>
> So, I extracted the std::string class and simplified it down and put it in
> another test case. That would be u.cpp and the results are results-u.txt. The
> results show the same performance trends although the absolute values have
> skewed. Will get back to this after I digest the results a bit more.

Attached some incomplete files previously. Corrected.

Liviu

Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Liviu Nicoara <ni...@hates.ms>.
On 10/3/12 11:10 AM, Martin Sebor wrote:
> On 10/03/2012 07:01 AM, Liviu Nicoara wrote:
>>
>> I am gathering some more measurements along these lines but it's time
>> consuming. I estimate I will have some ready for review later today or
>> tomorrow. In the meantime could you please post your kernel, glibc and
>> compiler versions?
>
> I was just thinking of a few simple loops along the lines of:
>
>    void* thread_func (void*) {
>        for (int i = 0; i < N; ++)
>            test 1: do some simple stuff inline
>            test 2: call a virtual function to do the same stuff
>            test 3: lock and unlock a mutex and do the same stuff
>    }
>
> Test 1 should be the fastest and test 3 the slowest. This should
> hold regardless of what "simple stuff" is (eventually, even when
> it's getting numpunct::grouping() data).

That is expected; I attached test case x.cpp and results-x.txt.

I did not find it too interesting in its own, though. The difference between the 
cached and non-cached data is that in the case of the cached data the copying of 
the string involves nothing more than a bump in the reference counter, whereas 
in the non-cached version a string object is constructed anew, and memory gets 
allocated for its body. Yet, in my measurements, the cached version is the one 
which shows the worse performance.

So, I extracted the std::string class and simplified it down and put it in 
another test case. That would be u.cpp and the results are results-u.txt. The 
results show the same performance trends although the absolute values have 
skewed. Will get back to this after I digest the results a bit more.

Liviu

Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Liviu Nicoara <ni...@hates.ms>.
On 10/26/12 18:50, Martin Sebor wrote:
> On 10/26/2012 06:50 AM, Liviu Nicoara wrote:
>> [...]
>> tl;dr: removing the facet data cache is a priority. All else can be put
>> on the back-burner.
>> [...]
>> // facet data accessor
>> ...
>> if (0 == _C_impsize) {           // 1
>>     mutex_lock ();
>>     if (_C_impsize)
>>         return _C_data;
>>     _C_data = get_facet_data (); // 2
>>     ??                           // 3
>>     _C_impsize = 1;              // 4
>>     mutex_unlock ();
>> }
>> ??                               // 5
>> return _C_data;                  // 6
>> [...]
>> Various compilers provide these features in various forms, but at the
>> moment we don't have a unified STDCXX API to implement this.
>>
>
> [...]
> I suggested moving the body of the outer if from the header
> into a .cpp file in the library where we could implement
> ugly, bloated locking without the risk of breaking things
> if we removed/replaced it in the future. That's when we ran
> into questions about how exactly to do this cleanly, etc.
> It didn't seem to be doable very cleanly but I still think
> it's a viable approach.

Just making sure we are talking about the same thing. My argument (and the code oversimplification above) was about the initialization of the std::facet data, not the cached numpunct data. I think we should get rid of the numpunct cache right away because it makes the facet unusable in MT builds.

As for the DCII I was talking about in the previous post, I would tackle it but I only have access to x86(_64). Does the foundation have shared dev machines?

Liviu

Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Martin Sebor <ms...@gmail.com>.
On 10/26/2012 06:50 AM, Liviu Nicoara wrote:
> On 10/03/12 11:10, Martin Sebor wrote:
>> [...]
>> I was just thinking of a few simple loops along the lines of:
>>
>> void* thread_func (void*) {
>> for (int i = 0; i < N; ++)
>> test 1: do some simple stuff inline
>> test 2: call a virtual function to do the same stuff
>> test 3: lock and unlock a mutex and do the same stuff
>> }
>>
>> Test 1 should be the fastest and test 3 the slowest. This should
>> hold regardless of what "simple stuff" is (eventually, even when
>> it's getting numpunct::grouping() data).
>
> tl;dr: removing the facet data cache is a priority. All else can be put
> on the back-burner.
>
> Conflicting test results aside, there still is the case of the incorrect
> handling of the cached data in the facet. I don't think there is a
> disagreement on that. Considering that the std::string is moving in the
> direction of dropping the handle-body implementation, simply getting rid
> of the cache is a step in the same direction.
>
> I think that we should preserve the lock-free reading of the facet data,
> as a benign race, but making it benign is perhaps more complicated than
> previously suggested.
>
> As a reminder, the core of the facet access and initialization code
> essentially looks like this (pseudocode-ish):
>
>
> // facet data accessor
> ...
> if (0 == _C_impsize) { // 1
> mutex_lock ();
> if (_C_impsize)
> return _C_data;
> _C_data = get_facet_data (); // 2
> ?? // 3
> _C_impsize = 1; // 4
> mutex_unlock ();
> }
> ?? // 5
> return _C_data; // 6
> ...
>
>
> with question marks for missing, necessary fixes. The compiler needs to
> be prevented from re-ordering both 2-4 and 1-6. Just for the sake of
> argument I can imagine an optimization that reorders the reads in 1-6:
>
> register x = _C_data;
> if (_C_impsize)
> return x;
>
> and if the loads are executed in this order, the caller will see a stale
> _C_data.
>
> First, the 2-4 writes need to be executed in the program order. This
> needs both a compiler barrier and a store-store memory barrier that will
> keep the writes ordered.
>
> Then, the reads in 1-6 need to be ordered such that _C_data is read
> after _C_impsize, via a compiler barrier and a load-load memory barrier
> that will preserve the program order of the loads.
>
> Various compilers provide these features in various forms, but at the
> moment we don't have a unified STDCXX API to implement this.
>
> Of course, I might be wrong. Input is appreciated.

You're right (we don't have an API for it, and it is the double
checked locking problem mentioned earlier on in this thread).
Implementing it here would bloat the function (just in terms
of source code, not necessarily object code), and potentially
introduce a binary compatibility dependency on the new code
(i.e., we wouldn't be able to remove the fix in the future
w/o braking it).

I suggested moving the body of the outer if from the header
into a .cpp file in the library where we could implement
ugly, bloated locking without the risk of breaking things
if we removed/replaced it in the future. That's when we ran
into questions about how exactly to do this cleanly, etc.
It didn't seem to be doable very cleanly but I still think
it's a viable approach.

Martin

>
> Thanks,
> Liviu
>


Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Liviu Nicoara <ni...@hates.ms>.
On 10/03/12 11:10, Martin Sebor wrote:
> [...]
> I was just thinking of a few simple loops along the lines of:
>
>    void* thread_func (void*) {
>        for (int i = 0; i < N; ++)
>            test 1: do some simple stuff inline
>            test 2: call a virtual function to do the same stuff
>            test 3: lock and unlock a mutex and do the same stuff
>    }
>
> Test 1 should be the fastest and test 3 the slowest. This should
> hold regardless of what "simple stuff" is (eventually, even when
> it's getting numpunct::grouping() data).

tl;dr: removing the facet data cache is a priority. All else can be put 
on the back-burner.

Conflicting test results aside, there still is the case of the incorrect 
handling of the cached data in the facet. I don't think there is a 
disagreement on that. Considering that the std::string is moving in the 
direction of dropping the handle-body implementation, simply getting rid 
of the cache is a step in the same direction.

I think that we should preserve the lock-free reading of the facet data, 
as a benign race, but making it benign is perhaps more complicated than 
previously suggested.

As a reminder, the core of the facet access and initialization code 
essentially looks like this (pseudocode-ish):


// facet data accessor
...
     if (0 == _C_impsize) {              // 1
         mutex_lock ();
         if (_C_impsize)
             return _C_data;
         _C_data    = get_facet_data (); // 2
         ??                              // 3
         _C_impsize = 1;                 // 4
         mutex_unlock ();
     }
     ??                                  // 5
     return _C_data;                     // 6
...


with question marks for missing, necessary fixes. The compiler needs to 
be prevented from re-ordering both 2-4 and 1-6. Just for the sake of 
argument I can imagine an optimization that reorders the reads in 1-6:

     register x = _C_data;
     if (_C_impsize)
         return x;

and if the loads are executed in this order, the caller will see a stale 
_C_data.

First, the 2-4 writes need to be executed in the program order. This 
needs both a compiler barrier and a store-store memory barrier that will 
keep the writes ordered.

Then, the reads in 1-6 need to be ordered such that _C_data is read 
after _C_impsize, via a compiler barrier and a load-load memory barrier 
that will preserve the program order of the loads.

Various compilers provide these features in various forms, but at the 
moment we don't have a unified STDCXX API to implement this.

Of course, I might be wrong. Input is appreciated.

Thanks,
Liviu


Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Martin Sebor <ms...@gmail.com>.
On 10/03/2012 07:01 AM, Liviu Nicoara wrote:
> On 10/02/12 10:41, Martin Sebor wrote:
>> I haven't had time to look at this since my last email on
>> Sunday. I also forgot about the string mutex. I don't think
>> I'll have time to spend on this until later in the week.
>> Unless the disassembly reveals the smoking gun, I think we
>> might need to simplify the test to get to the bottom of the
>> differences in our measurements. (I.e., eliminate the library
>> and measure the runtime of a simple thread loop, with and
>> without locking.) We should also look at the GLIBC and
>> kernel versions on our systems, on the off chance that
>> there has been a change that could explain the discrepancy
>> between my numbers and yours. I suspect my system (RHEL 4.8)
>> is much older than yours (I don't remember now if you posted
>> your details).
>
> I am gathering some more measurements along these lines but it's time
> consuming. I estimate I will have some ready for review later today or
> tomorrow. In the meantime could you please post your kernel, glibc and
> compiler versions?

I was just thinking of a few simple loops along the lines of:

   void* thread_func (void*) {
       for (int i = 0; i < N; ++)
           test 1: do some simple stuff inline
           test 2: call a virtual function to do the same stuff
           test 3: lock and unlock a mutex and do the same stuff
   }

Test 1 should be the fastest and test 3 the slowest. This should
hold regardless of what "simple stuff" is (eventually, even when
it's getting numpunct::grouping() data).

For the Linux tests I used a 16 CPU (Xeon X5570 @ 3GHz) box with
RHEL 4.8 with 2.6.9-89.0.11.ELlargesmp, GLIBC version is 2.3.4,
and GCC 3.4.6.

Martin

>
> Liviu
>
>>
>> Martin
>>
>> On 10/02/2012 06:22 AM, Liviu Nicoara wrote:
>>> On 09/30/12 18:18, Martin Sebor wrote:
>>>> I see you did a 64-bit build while I did a 32-bit one. so
>>>> I tried 64-bits. The cached version (i.e., the one compiled
>>>> with -UNO_USE_NUMPUNCT_CACHE) is still about twice as fast
>>>> as the non-cached one (compiled with -DNO_USE_NUMPUNCT_CACHE).
>>>>
>>>> I had made one change to the test program that I thought might
>>>> account for the difference: I removed the call to abort from
>>>> the thread function since it was causing the process to exit
>>>> prematurely in some of my tests. But since you used the
>>>> modified program for your latest measurements that couldn't
>>>> be it.
>>>>
>>>> I can't explain the differences. They just don't make sense
>>>> to me. Your results should be the other way around. Can you
>>>> post the disassembly of function f() for each of the two
>>>> configurations of the test?
>>>
>>> The first thing that struck me in the cached `f' was that __string_ref
>>> class uses a mutex for synchronizing access to the ref counter. It turns
>>> out, for Linux on AMD64 we explicitly use a mutex instead of the atomic
>>> ops on the ref counter, via a block in rw/_config.h:
>>>
>>> # if _RWSTD_VER_MAJOR < 5
>>> # ifdef _RWSTD_OS_LINUX
>>> // on Linux/AMD64, unless explicitly requested, disable the use
>>> // of atomic operations in string for binary compatibility with
>>> // stdcxx 4.1.x
>>> # ifndef _RWSTD_USE_STRING_ATOMIC_OPS
>>> # define _RWSTD_NO_STRING_ATOMIC_OPS
>>> # endif // _RWSTD_USE_STRING_ATOMIC_OPS
>>> # endif // _WIN32
>>> # endif // stdcxx < 5.0
>>>
>>>
>>> That is not the cause for the performance difference, though. Even after
>>> building with __RWSTD_USE_STRING_ATOMIC_OPS I get the same better
>>> performance with the non-cached version.
>>>
>>> Liviu
>


Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Liviu Nicoara <ni...@hates.ms>.
On 10/02/12 10:41, Martin Sebor wrote:
> I haven't had time to look at this since my last email on
> Sunday. I also forgot about the string mutex. I don't think
> I'll have time to spend on this until later in the week.
> Unless the disassembly reveals the smoking gun, I think we
> might need to simplify the test to get to the bottom of the
> differences in our measurements. (I.e., eliminate the library
> and measure the runtime of a simple thread loop, with and
> without locking.) We should also look at the GLIBC and
> kernel versions on our systems, on the off chance that
> there has been a change that could explain the discrepancy
> between my numbers and yours. I suspect my system (RHEL 4.8)
> is much older than yours (I don't remember now if you posted
> your details).

I am gathering some more measurements along these lines but it's time 
consuming. I estimate I will have some ready for review later today or 
tomorrow. In the meantime could you please post your kernel, glibc and 
compiler versions?

Liviu

>
> Martin
>
> On 10/02/2012 06:22 AM, Liviu Nicoara wrote:
>> On 09/30/12 18:18, Martin Sebor wrote:
>>> I see you did a 64-bit build while I did a 32-bit one. so
>>> I tried 64-bits. The cached version (i.e., the one compiled
>>> with -UNO_USE_NUMPUNCT_CACHE) is still about twice as fast
>>> as the non-cached one (compiled with -DNO_USE_NUMPUNCT_CACHE).
>>>
>>> I had made one change to the test program that I thought might
>>> account for the difference: I removed the call to abort from
>>> the thread function since it was causing the process to exit
>>> prematurely in some of my tests. But since you used the
>>> modified program for your latest measurements that couldn't
>>> be it.
>>>
>>> I can't explain the differences. They just don't make sense
>>> to me. Your results should be the other way around. Can you
>>> post the disassembly of function f() for each of the two
>>> configurations of the test?
>>
>> The first thing that struck me in the cached `f' was that __string_ref
>> class uses a mutex for synchronizing access to the ref counter. It turns
>> out, for Linux on AMD64 we explicitly use a mutex instead of the atomic
>> ops on the ref counter, via a block in rw/_config.h:
>>
>> # if _RWSTD_VER_MAJOR < 5
>> # ifdef _RWSTD_OS_LINUX
>> // on Linux/AMD64, unless explicitly requested, disable the use
>> // of atomic operations in string for binary compatibility with
>> // stdcxx 4.1.x
>> # ifndef _RWSTD_USE_STRING_ATOMIC_OPS
>> # define _RWSTD_NO_STRING_ATOMIC_OPS
>> # endif // _RWSTD_USE_STRING_ATOMIC_OPS
>> # endif // _WIN32
>> # endif // stdcxx < 5.0
>>
>>
>> That is not the cause for the performance difference, though. Even after
>> building with __RWSTD_USE_STRING_ATOMIC_OPS I get the same better
>> performance with the non-cached version.
>>
>> Liviu


Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Martin Sebor <ms...@gmail.com>.
I haven't had time to look at this since my last email on
Sunday. I also forgot about the string mutex. I don't think
I'll have time to spend on this until later in the week.
Unless the disassembly reveals the smoking gun, I think we
might need to simplify the test to get to the bottom of the
differences in our measurements. (I.e., eliminate the library
and measure the runtime of a simple thread loop, with and
without locking.) We should also look at the GLIBC and
kernel versions on our systems, on the off chance that
there has been a change that could explain the discrepancy
between my numbers and yours. I suspect my system (RHEL 4.8)
is much older than yours (I don't remember now if you posted
your details).

Martin

On 10/02/2012 06:22 AM, Liviu Nicoara wrote:
> On 09/30/12 18:18, Martin Sebor wrote:
>> I see you did a 64-bit build while I did a 32-bit one. so
>> I tried 64-bits. The cached version (i.e., the one compiled
>> with -UNO_USE_NUMPUNCT_CACHE) is still about twice as fast
>> as the non-cached one (compiled with -DNO_USE_NUMPUNCT_CACHE).
>>
>> I had made one change to the test program that I thought might
>> account for the difference: I removed the call to abort from
>> the thread function since it was causing the process to exit
>> prematurely in some of my tests. But since you used the
>> modified program for your latest measurements that couldn't
>> be it.
>>
>> I can't explain the differences. They just don't make sense
>> to me. Your results should be the other way around. Can you
>> post the disassembly of function f() for each of the two
>> configurations of the test?
>
> The first thing that struck me in the cached `f' was that __string_ref
> class uses a mutex for synchronizing access to the ref counter. It turns
> out, for Linux on AMD64 we explicitly use a mutex instead of the atomic
> ops on the ref counter, via a block in rw/_config.h:
>
> # if _RWSTD_VER_MAJOR < 5
> # ifdef _RWSTD_OS_LINUX
> // on Linux/AMD64, unless explicitly requested, disable the use
> // of atomic operations in string for binary compatibility with
> // stdcxx 4.1.x
> # ifndef _RWSTD_USE_STRING_ATOMIC_OPS
> # define _RWSTD_NO_STRING_ATOMIC_OPS
> # endif // _RWSTD_USE_STRING_ATOMIC_OPS
> # endif // _WIN32
> # endif // stdcxx < 5.0
>
>
> That is not the cause for the performance difference, though. Even after
> building with __RWSTD_USE_STRING_ATOMIC_OPS I get the same better
> performance with the non-cached version.
>
> Liviu


Re: Fwd: Re: STDCXX-1071 numpunct facet defect

Posted by Liviu Nicoara <ni...@hates.ms>.
On 09/30/12 18:18, Martin Sebor wrote:
> I see you did a 64-bit build while I did a 32-bit one. so
> I tried 64-bits. The cached version (i.e., the one compiled
> with -UNO_USE_NUMPUNCT_CACHE) is still about twice as fast
> as the non-cached one (compiled with -DNO_USE_NUMPUNCT_CACHE).
>
> I had made one change to the test program that I thought might
> account for the difference: I removed the call to abort from
> the thread function since it was causing the process to exit
> prematurely in some of my tests. But since you used the
> modified program for your latest measurements that couldn't
> be it.
>
> I can't explain the differences. They just don't make sense
> to me. Your results should be the other way around. Can you
> post the disassembly of function f() for each of the two
> configurations of the test?

The first thing that struck me in the cached `f' was that __string_ref 
class uses a mutex for synchronizing access to the ref counter. It turns 
out, for Linux on AMD64 we explicitly use a mutex instead of the atomic 
ops on the ref counter, via a block in rw/_config.h:

#  if _RWSTD_VER_MAJOR < 5
#    ifdef _RWSTD_OS_LINUX
        // on Linux/AMD64, unless explicitly requested, disable the use
        // of atomic operations in string for binary compatibility with
        // stdcxx 4.1.x
#      ifndef _RWSTD_USE_STRING_ATOMIC_OPS
#        define _RWSTD_NO_STRING_ATOMIC_OPS
#      endif   // _RWSTD_USE_STRING_ATOMIC_OPS
#    endif   // _WIN32
#  endif   // stdcxx < 5.0


That is not the cause for the performance difference, though. Even after 
building with __RWSTD_USE_STRING_ATOMIC_OPS I get the same better 
performance with the non-cached version.

Liviu