You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Ergin Ozekes <er...@gmail.com> on 2014/10/23 11:33:49 UTC
packets collapsed in receive queue due to low socket buffer
Hi all;
My current working network bandwidth is 350 Mbps.
My netstat -st output prints out below value increasingly.
1555903887 packets collapsed in receive queue due to low socket buffer
1555908175 packets collapsed in receive queue due to low socket buffer
1555912925 packets collapsed in receive queue due to low socket buffer
1555920054 packets collapsed in receive queue due to low socket buffer
1555929162 packets collapsed in receive queue due to low socket buffer
1555938162 packets collapsed in receive queue due to low socket buffer
1555945682 packets collapsed in receive queue due to low socket buffer
1555951783 packets collapsed in receive queue due to low socket buffer
1555959318 packets collapsed in receive queue due to low socket buffer
1555962474 packets collapsed in receive queue due to low socket buffer
1555969574 packets collapsed in receive queue due to low socket buffer
I've increased to socket buffer size, backlog queue size and memlock value
was set to unlimited for ats user. How can I fix it completely?
BTW, How can I increase read from socket(connection) performance of the ATS?
Best Regards,
Ergin Ozekes
On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes <er...@gmail.com>
wrote:
> Hi all;
>
> My current working network bandwidth is 350 Mbps.
> My netstat -st output prints out below value increasingly.
>
> 1555903887 packets collapsed in receive queue due to low socket buffer
> 1555908175 packets collapsed in receive queue due to low socket buffer
> 1555912925 packets collapsed in receive queue due to low socket buffer
> 1555920054 packets collapsed in receive queue due to low socket buffer
> 1555929162 packets collapsed in receive queue due to low socket buffer
> 1555938162 packets collapsed in receive queue due to low socket buffer
> 1555945682 packets collapsed in receive queue due to low socket buffer
> 1555951783 packets collapsed in receive queue due to low socket buffer
> 1555959318 packets collapsed in receive queue due to low socket buffer
> 1555962474 packets collapsed in receive queue due to low socket buffer
> 1555969574 packets collapsed in receive queue due to low socket buffer
>
> I've increased to socket buffer size, backlog queue size and memlock value
> was set to unlimited for ats user. How can I fix it completely?
>
> Best Regards,
> Ergin Ozekes
>
Re: packets collapsed in receive queue due to low socket buffer
Posted by Ergin Ozekes <er...@gmail.com>.
Hi Ben;
In according to your feedback about autosizing I've change my setting as
below:
net.core.rmem_default = 536870912
net.core.wmem_default = 536870912
net.core.rmem_max = 536870912
net.core.wmem_max = 536870912
net.ipv4.tcp_mem = 10240 87380 268435456
net.ipv4.tcp_rmem = 10240 87380 268435456
net.ipv4.tcp_wmem = 10240 87380 268435456
My ATS usage case is forward transparent cache. In according to my rtt
values and network traffic(350 Mbps) buffer size is 256Mbps (=rtt*bw).
> I've increased the some of the queue size as is below:
> net.core.netdev_max_backlog = 300000
> net.core.somaxconn = 65535
> net.ipv4.tcp_max_orphans = 268435456
> net.ipv4.tcp_max_syn_backlog = 268435456
> > > CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760
> > > CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760
> > > CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760
> > > CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760
I've used above values to prevent bad segment received error on the output
of "netstat -s". This is the my nightmare. The "packets collapsed in
receive queue" and "bad segment receive" issues are causing the netwotk
traffic increasing on the ats-out traffic. Because there is too much packet
retransmitions.
Do you have any idea to fix this retransmissions?
Best Regards,
Ergin
On Sun, Oct 26, 2014 at 12:29 PM, Ben <be...@meh.net.nz> wrote:
> On Sun, Oct 26, 2014 at 11:05:51AM +0200, Ergin Ozekes wrote:
> > Hi james;
> >
> > In the beginning of my investigation I've set the all interfaces ring
> > buffer size max value of the ethernet cards ( which is 4078 by using
> > ethtool -g). And also I've increased the socket buffer size by using
> kernel
> > parameters as is below.
> > net.core.rmem_default=536870912
> > net.core.wmem_default=536870912
> > net.core.rmem_max=536870912
> > net.core.wmem_max=536870912
> > net.ipv4.tcp_mem=268435456 268435456 268435456
> > net.ipv4.tcp_rmem=268435456 268435456 268435456
> > net.ipv4.tcp_wmem=268435456 268435456 268435456
>
> You really don't need to do that, and it could make things worse rather
> than better.
>
> > I've increased the some of the queue size as is below:
> > net.core.netdev_max_backlog = 300000
> > net.core.somaxconn = 65535
> > net.ipv4.tcp_max_orphans = 268435456
> > net.ipv4.tcp_max_syn_backlog = 268435456
>
>
> Why raise these so high?
>
> > Also kernel shared memory size.
> >
> > kernel.shmmax = 16000000000
> > kernel.shmall = 4000000000
> >
> > After these setting I saw the increasing "segment retransmited" and
> > "packets collapsed in receive queue due to low socket buffer" lines on
> the
> > "netstat -st" output.
>
> Probably because you're running out of RAM.
>
> net.ipv4.tcp_rmem, and net.ipv4.tcp_wmem are per connection.
>
> autosizing mostly works well, take advantage of it.
>
> > I think there are timeout values or other parameter settings which
> triggers
> > the above failures.
> >
> > Do you have any idea?
> > Best Regards,
> > Ergin
> >
> >
> > On Sat, Oct 25, 2014 at 1:46 AM, James Peach <jp...@apache.org> wrote:
> >
> > >
> > > > On Oct 24, 2014, at 12:52 AM, Ergin Ozekes <er...@gmail.com>
> > > wrote:
> > > >
> > > > Hi All;
> > > >
> > > > My problem's solution is as below:
> > > >
> > > >
> > >
> ##############################################################################
> > > > # Buffer size
> > > >
> > >
> ##############################################################################
> > > > CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760
> > > > CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760
> > > > CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760
> > > > CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760
>
> So you're saying that you need 10 megabytes buffer size on input and
> output to server and client.
>
> I can do over 10 gigabit/sec with a single client with 512k buffer easily
> on LAN. You only need high
> values if you have a really fast connection with a really high latency.
> Like 10 megabytes at 100 msec ping
> would be 100 megabytes/sec to a single client.
>
> For normal web loads window sizes never even get very high if you use
> autosizing as most objects are small
> etc.
>
> > > Great! You might want to check the buffers on your network adaptors as
> > > well (ethtool -g IIRC)
> > >
> > > >
> > > > Best Regards,
> > > > Ergin
> > > >
> > > > On Thu, Oct 23, 2014 at 6:33 PM, Ergin Ozekes <
> ergin.ozekes@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi all;
> > > >>
> > > >> In according to my investigation on the ats read
> > > >> performance,NetHandler::mainNetEvent is the top function. On my
> system
> > > >> lots of buffer over run is exist. In my system net input interface
> has
> > > 250
> > > >> Mbps traffic but on the client output only 125 Mbps is exist. Half
> of
> > > the
> > > >> downloaded content is returning to client.
> > > >>
> > > >> Does anyone knows the reason of this case?
> > > >> And how can I fixe it?
> > > >>
> > > >> This is the "perf top" output:
> > > >> 4.82% libpcre.so.3.13.1 [.] match
> > > >>
> > > >>
> > > >> 3.27% [kernel] [k] native_write_msr_safe
> > > >>
> > > >>
> > > >> 2.09% [kernel] [k] ipt_do_table
> > > >>
> > > >>
> > > >> 1.60% [kernel] [k] __schedule
> > > >>
> > > >>
> > > >> 1.57% [kernel] [k] menu_select
> > > >>
> > > >>
> > > >> 1.43% libc-2.13.so [.] 0x00000000000e92b0
> > > >>
> > > >>
> > > >> 1.35% [kernel] [k] find_busiest_group
> > > >>
> > > >>
> > > >> 1.16% traffic_server [.] EThread::execute()
> > > >>
> > > >>
> > > >> 1.12% [kernel] [k] copy_user_generic_string
> > > >>
> > > >>
> > > >> 1.10% [kernel] [k] nf_iterate
> > > >>
> > > >>
> > > >> 1.04% [kernel] [k] _raw_spin_lock_irqsave
> > > >>
> > > >>
> > > >> 1.03% [kernel] [k] int_sqrt
> > > >>
> > > >>
> > > >> *1.02% traffic_server [.] NetHandler::mainNetEvent(int,
> > > >> Event*)*
> > > >>
> > > >> 0.96% [bnx2x] [k] bnx2x_rx_int
> > > >>
> > > >>
> > > >> 0.93% [kernel] [k] _raw_spin_lock
> > > >>
> > > >>
> > > >> 0.90% [kernel] [k] htable_selective_cleanup
> > > >>
> > > >>
> > > >> 0.86% [kernel] [k] cpuidle_enter_state
> > > >>
> > > >>
> > > >> 0.83% [kernel] [k] enqueue_task_fair
> > > >>
> > > >>
> > > >> 0.83% [kernel] [k] tcp_packet
> > > >>
> > > >>
> > > >> 0.76% [kernel] [k] apic_timer_interrupt
> > > >>
> > > >>
> > > >> 0.76% [kernel] [k] timerqueue_add
> > > >>
> > > >>
> > > >> 0.76% [kernel] [k] idle_cpu
> > > >>
> > > >>
> > > >> 0.71% [bnx2x] [k] bnx2x_start_xmit
> > > >>
> > > >>
> > > >> 0.67% [kernel] [k] rb_erase
> > > >>
> > > >>
> > > >> *0.64% traffic_server [.] read_from_net(NetHandler*,
> > > >> UnixNetVConnection*, EThread*) *
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Oct 23, 2014 at 12:33 PM, Ergin Ozekes <
> ergin.ozekes@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi all;
> > > >>>
> > > >>> My current working network bandwidth is 350 Mbps.
> > > >>> My netstat -st output prints out below value increasingly.
> > > >>>
> > > >>> 1555903887 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555908175 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555912925 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555920054 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555929162 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555938162 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555945682 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555951783 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555959318 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555962474 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>> 1555969574 packets collapsed in receive queue due to low socket
> > > buffer
> > > >>>
> > > >>> I've increased to socket buffer size, backlog queue size and
> memlock
> > > >>> value was set to unlimited for ats user. How can I fix it
> completely?
> > > >>> BTW, How can I increase read from socket(connection) performance
> of the
> > > >>> ATS?
> > > >>>
> > > >>>
> > > >>> Best Regards,
> > > >>> Ergin Ozekes
> > > >>>
> > > >>> On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes <
> ergin.ozekes@gmail.com
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> Hi all;
> > > >>>>
> > > >>>> My current working network bandwidth is 350 Mbps.
> > > >>>> My netstat -st output prints out below value increasingly.
> > > >>>>
> > > >>>> 1555903887 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555908175 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555912925 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555920054 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555929162 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555938162 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555945682 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555951783 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555959318 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555962474 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>> 1555969574 packets collapsed in receive queue due to low socket
> > > >>>> buffer
> > > >>>>
> > > >>>> I've increased to socket buffer size, backlog queue size and
> memlock
> > > >>>> value was set to unlimited for ats user. How can I fix it
> completely?
> > > >>>>
> > > >>>> Best Regards,
> > > >>>> Ergin Ozekes
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
>
Re: packets collapsed in receive queue due to low socket buffer
Posted by Ben <be...@meh.net.nz>.
On Sun, Oct 26, 2014 at 11:05:51AM +0200, Ergin Ozekes wrote:
> Hi james;
>
> In the beginning of my investigation I've set the all interfaces ring
> buffer size max value of the ethernet cards ( which is 4078 by using
> ethtool -g). And also I've increased the socket buffer size by using kernel
> parameters as is below.
> net.core.rmem_default=536870912
> net.core.wmem_default=536870912
> net.core.rmem_max=536870912
> net.core.wmem_max=536870912
> net.ipv4.tcp_mem=268435456 268435456 268435456
> net.ipv4.tcp_rmem=268435456 268435456 268435456
> net.ipv4.tcp_wmem=268435456 268435456 268435456
You really don't need to do that, and it could make things worse rather than better.
> I've increased the some of the queue size as is below:
> net.core.netdev_max_backlog = 300000
> net.core.somaxconn = 65535
> net.ipv4.tcp_max_orphans = 268435456
> net.ipv4.tcp_max_syn_backlog = 268435456
Why raise these so high?
> Also kernel shared memory size.
>
> kernel.shmmax = 16000000000
> kernel.shmall = 4000000000
>
> After these setting I saw the increasing "segment retransmited" and
> "packets collapsed in receive queue due to low socket buffer" lines on the
> "netstat -st" output.
Probably because you're running out of RAM.
net.ipv4.tcp_rmem, and net.ipv4.tcp_wmem are per connection.
autosizing mostly works well, take advantage of it.
> I think there are timeout values or other parameter settings which triggers
> the above failures.
>
> Do you have any idea?
> Best Regards,
> Ergin
>
>
> On Sat, Oct 25, 2014 at 1:46 AM, James Peach <jp...@apache.org> wrote:
>
> >
> > > On Oct 24, 2014, at 12:52 AM, Ergin Ozekes <er...@gmail.com>
> > wrote:
> > >
> > > Hi All;
> > >
> > > My problem's solution is as below:
> > >
> > >
> > ##############################################################################
> > > # Buffer size
> > >
> > ##############################################################################
> > > CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760
> > > CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760
> > > CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760
> > > CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760
So you're saying that you need 10 megabytes buffer size on input and output to server and client.
I can do over 10 gigabit/sec with a single client with 512k buffer easily on LAN. You only need high
values if you have a really fast connection with a really high latency. Like 10 megabytes at 100 msec ping
would be 100 megabytes/sec to a single client.
For normal web loads window sizes never even get very high if you use autosizing as most objects are small
etc.
> > Great! You might want to check the buffers on your network adaptors as
> > well (ethtool -g IIRC)
> >
> > >
> > > Best Regards,
> > > Ergin
> > >
> > > On Thu, Oct 23, 2014 at 6:33 PM, Ergin Ozekes <er...@gmail.com>
> > > wrote:
> > >
> > >> Hi all;
> > >>
> > >> In according to my investigation on the ats read
> > >> performance,NetHandler::mainNetEvent is the top function. On my system
> > >> lots of buffer over run is exist. In my system net input interface has
> > 250
> > >> Mbps traffic but on the client output only 125 Mbps is exist. Half of
> > the
> > >> downloaded content is returning to client.
> > >>
> > >> Does anyone knows the reason of this case?
> > >> And how can I fixe it?
> > >>
> > >> This is the "perf top" output:
> > >> 4.82% libpcre.so.3.13.1 [.] match
> > >>
> > >>
> > >> 3.27% [kernel] [k] native_write_msr_safe
> > >>
> > >>
> > >> 2.09% [kernel] [k] ipt_do_table
> > >>
> > >>
> > >> 1.60% [kernel] [k] __schedule
> > >>
> > >>
> > >> 1.57% [kernel] [k] menu_select
> > >>
> > >>
> > >> 1.43% libc-2.13.so [.] 0x00000000000e92b0
> > >>
> > >>
> > >> 1.35% [kernel] [k] find_busiest_group
> > >>
> > >>
> > >> 1.16% traffic_server [.] EThread::execute()
> > >>
> > >>
> > >> 1.12% [kernel] [k] copy_user_generic_string
> > >>
> > >>
> > >> 1.10% [kernel] [k] nf_iterate
> > >>
> > >>
> > >> 1.04% [kernel] [k] _raw_spin_lock_irqsave
> > >>
> > >>
> > >> 1.03% [kernel] [k] int_sqrt
> > >>
> > >>
> > >> *1.02% traffic_server [.] NetHandler::mainNetEvent(int,
> > >> Event*)*
> > >>
> > >> 0.96% [bnx2x] [k] bnx2x_rx_int
> > >>
> > >>
> > >> 0.93% [kernel] [k] _raw_spin_lock
> > >>
> > >>
> > >> 0.90% [kernel] [k] htable_selective_cleanup
> > >>
> > >>
> > >> 0.86% [kernel] [k] cpuidle_enter_state
> > >>
> > >>
> > >> 0.83% [kernel] [k] enqueue_task_fair
> > >>
> > >>
> > >> 0.83% [kernel] [k] tcp_packet
> > >>
> > >>
> > >> 0.76% [kernel] [k] apic_timer_interrupt
> > >>
> > >>
> > >> 0.76% [kernel] [k] timerqueue_add
> > >>
> > >>
> > >> 0.76% [kernel] [k] idle_cpu
> > >>
> > >>
> > >> 0.71% [bnx2x] [k] bnx2x_start_xmit
> > >>
> > >>
> > >> 0.67% [kernel] [k] rb_erase
> > >>
> > >>
> > >> *0.64% traffic_server [.] read_from_net(NetHandler*,
> > >> UnixNetVConnection*, EThread*) *
> > >>
> > >>
> > >>
> > >> On Thu, Oct 23, 2014 at 12:33 PM, Ergin Ozekes <er...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi all;
> > >>>
> > >>> My current working network bandwidth is 350 Mbps.
> > >>> My netstat -st output prints out below value increasingly.
> > >>>
> > >>> 1555903887 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555908175 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555912925 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555920054 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555929162 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555938162 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555945682 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555951783 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555959318 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555962474 packets collapsed in receive queue due to low socket
> > buffer
> > >>> 1555969574 packets collapsed in receive queue due to low socket
> > buffer
> > >>>
> > >>> I've increased to socket buffer size, backlog queue size and memlock
> > >>> value was set to unlimited for ats user. How can I fix it completely?
> > >>> BTW, How can I increase read from socket(connection) performance of the
> > >>> ATS?
> > >>>
> > >>>
> > >>> Best Regards,
> > >>> Ergin Ozekes
> > >>>
> > >>> On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes <ergin.ozekes@gmail.com
> > >
> > >>> wrote:
> > >>>
> > >>>> Hi all;
> > >>>>
> > >>>> My current working network bandwidth is 350 Mbps.
> > >>>> My netstat -st output prints out below value increasingly.
> > >>>>
> > >>>> 1555903887 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555908175 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555912925 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555920054 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555929162 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555938162 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555945682 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555951783 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555959318 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555962474 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>> 1555969574 packets collapsed in receive queue due to low socket
> > >>>> buffer
> > >>>>
> > >>>> I've increased to socket buffer size, backlog queue size and memlock
> > >>>> value was set to unlimited for ats user. How can I fix it completely?
> > >>>>
> > >>>> Best Regards,
> > >>>> Ergin Ozekes
> > >>>>
> > >>>
> > >>>
> > >>
> >
> >
Re: packets collapsed in receive queue due to low socket buffer
Posted by Ergin Ozekes <er...@gmail.com>.
Hi james;
In the beginning of my investigation I've set the all interfaces ring
buffer size max value of the ethernet cards ( which is 4078 by using
ethtool -g). And also I've increased the socket buffer size by using kernel
parameters as is below.
net.core.rmem_default=536870912
net.core.wmem_default=536870912
net.core.rmem_max=536870912
net.core.wmem_max=536870912
net.ipv4.tcp_mem=268435456 268435456 268435456
net.ipv4.tcp_rmem=268435456 268435456 268435456
net.ipv4.tcp_wmem=268435456 268435456 268435456
I've increased the some of the queue size as is below:
net.core.netdev_max_backlog = 300000
net.core.somaxconn = 65535
net.ipv4.tcp_max_orphans = 268435456
net.ipv4.tcp_max_syn_backlog = 268435456
Also kernel shared memory size.
kernel.shmmax = 16000000000
kernel.shmall = 4000000000
After these setting I saw the increasing "segment retransmited" and
"packets collapsed in receive queue due to low socket buffer" lines on the
"netstat -st" output.
I think there are timeout values or other parameter settings which triggers
the above failures.
Do you have any idea?
Best Regards,
Ergin
On Sat, Oct 25, 2014 at 1:46 AM, James Peach <jp...@apache.org> wrote:
>
> > On Oct 24, 2014, at 12:52 AM, Ergin Ozekes <er...@gmail.com>
> wrote:
> >
> > Hi All;
> >
> > My problem's solution is as below:
> >
> >
> ##############################################################################
> > # Buffer size
> >
> ##############################################################################
> > CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760
> > CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760
> > CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760
> > CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760
>
> Great! You might want to check the buffers on your network adaptors as
> well (ethtool -g IIRC)
>
> >
> > Best Regards,
> > Ergin
> >
> > On Thu, Oct 23, 2014 at 6:33 PM, Ergin Ozekes <er...@gmail.com>
> > wrote:
> >
> >> Hi all;
> >>
> >> In according to my investigation on the ats read
> >> performance,NetHandler::mainNetEvent is the top function. On my system
> >> lots of buffer over run is exist. In my system net input interface has
> 250
> >> Mbps traffic but on the client output only 125 Mbps is exist. Half of
> the
> >> downloaded content is returning to client.
> >>
> >> Does anyone knows the reason of this case?
> >> And how can I fixe it?
> >>
> >> This is the "perf top" output:
> >> 4.82% libpcre.so.3.13.1 [.] match
> >>
> >>
> >> 3.27% [kernel] [k] native_write_msr_safe
> >>
> >>
> >> 2.09% [kernel] [k] ipt_do_table
> >>
> >>
> >> 1.60% [kernel] [k] __schedule
> >>
> >>
> >> 1.57% [kernel] [k] menu_select
> >>
> >>
> >> 1.43% libc-2.13.so [.] 0x00000000000e92b0
> >>
> >>
> >> 1.35% [kernel] [k] find_busiest_group
> >>
> >>
> >> 1.16% traffic_server [.] EThread::execute()
> >>
> >>
> >> 1.12% [kernel] [k] copy_user_generic_string
> >>
> >>
> >> 1.10% [kernel] [k] nf_iterate
> >>
> >>
> >> 1.04% [kernel] [k] _raw_spin_lock_irqsave
> >>
> >>
> >> 1.03% [kernel] [k] int_sqrt
> >>
> >>
> >> *1.02% traffic_server [.] NetHandler::mainNetEvent(int,
> >> Event*)*
> >>
> >> 0.96% [bnx2x] [k] bnx2x_rx_int
> >>
> >>
> >> 0.93% [kernel] [k] _raw_spin_lock
> >>
> >>
> >> 0.90% [kernel] [k] htable_selective_cleanup
> >>
> >>
> >> 0.86% [kernel] [k] cpuidle_enter_state
> >>
> >>
> >> 0.83% [kernel] [k] enqueue_task_fair
> >>
> >>
> >> 0.83% [kernel] [k] tcp_packet
> >>
> >>
> >> 0.76% [kernel] [k] apic_timer_interrupt
> >>
> >>
> >> 0.76% [kernel] [k] timerqueue_add
> >>
> >>
> >> 0.76% [kernel] [k] idle_cpu
> >>
> >>
> >> 0.71% [bnx2x] [k] bnx2x_start_xmit
> >>
> >>
> >> 0.67% [kernel] [k] rb_erase
> >>
> >>
> >> *0.64% traffic_server [.] read_from_net(NetHandler*,
> >> UnixNetVConnection*, EThread*) *
> >>
> >>
> >>
> >> On Thu, Oct 23, 2014 at 12:33 PM, Ergin Ozekes <er...@gmail.com>
> >> wrote:
> >>
> >>> Hi all;
> >>>
> >>> My current working network bandwidth is 350 Mbps.
> >>> My netstat -st output prints out below value increasingly.
> >>>
> >>> 1555903887 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555908175 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555912925 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555920054 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555929162 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555938162 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555945682 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555951783 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555959318 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555962474 packets collapsed in receive queue due to low socket
> buffer
> >>> 1555969574 packets collapsed in receive queue due to low socket
> buffer
> >>>
> >>> I've increased to socket buffer size, backlog queue size and memlock
> >>> value was set to unlimited for ats user. How can I fix it completely?
> >>> BTW, How can I increase read from socket(connection) performance of the
> >>> ATS?
> >>>
> >>>
> >>> Best Regards,
> >>> Ergin Ozekes
> >>>
> >>> On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes <ergin.ozekes@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Hi all;
> >>>>
> >>>> My current working network bandwidth is 350 Mbps.
> >>>> My netstat -st output prints out below value increasingly.
> >>>>
> >>>> 1555903887 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555908175 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555912925 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555920054 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555929162 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555938162 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555945682 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555951783 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555959318 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555962474 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>> 1555969574 packets collapsed in receive queue due to low socket
> >>>> buffer
> >>>>
> >>>> I've increased to socket buffer size, backlog queue size and memlock
> >>>> value was set to unlimited for ats user. How can I fix it completely?
> >>>>
> >>>> Best Regards,
> >>>> Ergin Ozekes
> >>>>
> >>>
> >>>
> >>
>
>
Re: packets collapsed in receive queue due to low socket buffer
Posted by James Peach <jp...@apache.org>.
> On Oct 24, 2014, at 12:52 AM, Ergin Ozekes <er...@gmail.com> wrote:
>
> Hi All;
>
> My problem's solution is as below:
>
> ##############################################################################
> # Buffer size
> ##############################################################################
> CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760
> CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760
> CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760
> CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760
Great! You might want to check the buffers on your network adaptors as well (ethtool -g IIRC)
>
> Best Regards,
> Ergin
>
> On Thu, Oct 23, 2014 at 6:33 PM, Ergin Ozekes <er...@gmail.com>
> wrote:
>
>> Hi all;
>>
>> In according to my investigation on the ats read
>> performance,NetHandler::mainNetEvent is the top function. On my system
>> lots of buffer over run is exist. In my system net input interface has 250
>> Mbps traffic but on the client output only 125 Mbps is exist. Half of the
>> downloaded content is returning to client.
>>
>> Does anyone knows the reason of this case?
>> And how can I fixe it?
>>
>> This is the "perf top" output:
>> 4.82% libpcre.so.3.13.1 [.] match
>>
>>
>> 3.27% [kernel] [k] native_write_msr_safe
>>
>>
>> 2.09% [kernel] [k] ipt_do_table
>>
>>
>> 1.60% [kernel] [k] __schedule
>>
>>
>> 1.57% [kernel] [k] menu_select
>>
>>
>> 1.43% libc-2.13.so [.] 0x00000000000e92b0
>>
>>
>> 1.35% [kernel] [k] find_busiest_group
>>
>>
>> 1.16% traffic_server [.] EThread::execute()
>>
>>
>> 1.12% [kernel] [k] copy_user_generic_string
>>
>>
>> 1.10% [kernel] [k] nf_iterate
>>
>>
>> 1.04% [kernel] [k] _raw_spin_lock_irqsave
>>
>>
>> 1.03% [kernel] [k] int_sqrt
>>
>>
>> *1.02% traffic_server [.] NetHandler::mainNetEvent(int,
>> Event*)*
>>
>> 0.96% [bnx2x] [k] bnx2x_rx_int
>>
>>
>> 0.93% [kernel] [k] _raw_spin_lock
>>
>>
>> 0.90% [kernel] [k] htable_selective_cleanup
>>
>>
>> 0.86% [kernel] [k] cpuidle_enter_state
>>
>>
>> 0.83% [kernel] [k] enqueue_task_fair
>>
>>
>> 0.83% [kernel] [k] tcp_packet
>>
>>
>> 0.76% [kernel] [k] apic_timer_interrupt
>>
>>
>> 0.76% [kernel] [k] timerqueue_add
>>
>>
>> 0.76% [kernel] [k] idle_cpu
>>
>>
>> 0.71% [bnx2x] [k] bnx2x_start_xmit
>>
>>
>> 0.67% [kernel] [k] rb_erase
>>
>>
>> *0.64% traffic_server [.] read_from_net(NetHandler*,
>> UnixNetVConnection*, EThread*) *
>>
>>
>>
>> On Thu, Oct 23, 2014 at 12:33 PM, Ergin Ozekes <er...@gmail.com>
>> wrote:
>>
>>> Hi all;
>>>
>>> My current working network bandwidth is 350 Mbps.
>>> My netstat -st output prints out below value increasingly.
>>>
>>> 1555903887 packets collapsed in receive queue due to low socket buffer
>>> 1555908175 packets collapsed in receive queue due to low socket buffer
>>> 1555912925 packets collapsed in receive queue due to low socket buffer
>>> 1555920054 packets collapsed in receive queue due to low socket buffer
>>> 1555929162 packets collapsed in receive queue due to low socket buffer
>>> 1555938162 packets collapsed in receive queue due to low socket buffer
>>> 1555945682 packets collapsed in receive queue due to low socket buffer
>>> 1555951783 packets collapsed in receive queue due to low socket buffer
>>> 1555959318 packets collapsed in receive queue due to low socket buffer
>>> 1555962474 packets collapsed in receive queue due to low socket buffer
>>> 1555969574 packets collapsed in receive queue due to low socket buffer
>>>
>>> I've increased to socket buffer size, backlog queue size and memlock
>>> value was set to unlimited for ats user. How can I fix it completely?
>>> BTW, How can I increase read from socket(connection) performance of the
>>> ATS?
>>>
>>>
>>> Best Regards,
>>> Ergin Ozekes
>>>
>>> On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes <er...@gmail.com>
>>> wrote:
>>>
>>>> Hi all;
>>>>
>>>> My current working network bandwidth is 350 Mbps.
>>>> My netstat -st output prints out below value increasingly.
>>>>
>>>> 1555903887 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555908175 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555912925 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555920054 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555929162 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555938162 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555945682 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555951783 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555959318 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555962474 packets collapsed in receive queue due to low socket
>>>> buffer
>>>> 1555969574 packets collapsed in receive queue due to low socket
>>>> buffer
>>>>
>>>> I've increased to socket buffer size, backlog queue size and memlock
>>>> value was set to unlimited for ats user. How can I fix it completely?
>>>>
>>>> Best Regards,
>>>> Ergin Ozekes
>>>>
>>>
>>>
>>
Re: packets collapsed in receive queue due to low socket buffer
Posted by Ergin Ozekes <er...@gmail.com>.
Hi All;
My problem's solution is as below:
##############################################################################
# Buffer size
##############################################################################
CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760
CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760
CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760
CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760
Best Regards,
Ergin
On Thu, Oct 23, 2014 at 6:33 PM, Ergin Ozekes <er...@gmail.com>
wrote:
> Hi all;
>
> In according to my investigation on the ats read
> performance,NetHandler::mainNetEvent is the top function. On my system
> lots of buffer over run is exist. In my system net input interface has 250
> Mbps traffic but on the client output only 125 Mbps is exist. Half of the
> downloaded content is returning to client.
>
> Does anyone knows the reason of this case?
> And how can I fixe it?
>
> This is the "perf top" output:
> 4.82% libpcre.so.3.13.1 [.] match
>
>
> 3.27% [kernel] [k] native_write_msr_safe
>
>
> 2.09% [kernel] [k] ipt_do_table
>
>
> 1.60% [kernel] [k] __schedule
>
>
> 1.57% [kernel] [k] menu_select
>
>
> 1.43% libc-2.13.so [.] 0x00000000000e92b0
>
>
> 1.35% [kernel] [k] find_busiest_group
>
>
> 1.16% traffic_server [.] EThread::execute()
>
>
> 1.12% [kernel] [k] copy_user_generic_string
>
>
> 1.10% [kernel] [k] nf_iterate
>
>
> 1.04% [kernel] [k] _raw_spin_lock_irqsave
>
>
> 1.03% [kernel] [k] int_sqrt
>
>
> *1.02% traffic_server [.] NetHandler::mainNetEvent(int,
> Event*)*
>
> 0.96% [bnx2x] [k] bnx2x_rx_int
>
>
> 0.93% [kernel] [k] _raw_spin_lock
>
>
> 0.90% [kernel] [k] htable_selective_cleanup
>
>
> 0.86% [kernel] [k] cpuidle_enter_state
>
>
> 0.83% [kernel] [k] enqueue_task_fair
>
>
> 0.83% [kernel] [k] tcp_packet
>
>
> 0.76% [kernel] [k] apic_timer_interrupt
>
>
> 0.76% [kernel] [k] timerqueue_add
>
>
> 0.76% [kernel] [k] idle_cpu
>
>
> 0.71% [bnx2x] [k] bnx2x_start_xmit
>
>
> 0.67% [kernel] [k] rb_erase
>
>
> *0.64% traffic_server [.] read_from_net(NetHandler*,
> UnixNetVConnection*, EThread*) *
>
>
>
> On Thu, Oct 23, 2014 at 12:33 PM, Ergin Ozekes <er...@gmail.com>
> wrote:
>
>> Hi all;
>>
>> My current working network bandwidth is 350 Mbps.
>> My netstat -st output prints out below value increasingly.
>>
>> 1555903887 packets collapsed in receive queue due to low socket buffer
>> 1555908175 packets collapsed in receive queue due to low socket buffer
>> 1555912925 packets collapsed in receive queue due to low socket buffer
>> 1555920054 packets collapsed in receive queue due to low socket buffer
>> 1555929162 packets collapsed in receive queue due to low socket buffer
>> 1555938162 packets collapsed in receive queue due to low socket buffer
>> 1555945682 packets collapsed in receive queue due to low socket buffer
>> 1555951783 packets collapsed in receive queue due to low socket buffer
>> 1555959318 packets collapsed in receive queue due to low socket buffer
>> 1555962474 packets collapsed in receive queue due to low socket buffer
>> 1555969574 packets collapsed in receive queue due to low socket buffer
>>
>> I've increased to socket buffer size, backlog queue size and memlock
>> value was set to unlimited for ats user. How can I fix it completely?
>> BTW, How can I increase read from socket(connection) performance of the
>> ATS?
>>
>>
>> Best Regards,
>> Ergin Ozekes
>>
>> On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes <er...@gmail.com>
>> wrote:
>>
>>> Hi all;
>>>
>>> My current working network bandwidth is 350 Mbps.
>>> My netstat -st output prints out below value increasingly.
>>>
>>> 1555903887 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555908175 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555912925 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555920054 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555929162 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555938162 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555945682 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555951783 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555959318 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555962474 packets collapsed in receive queue due to low socket
>>> buffer
>>> 1555969574 packets collapsed in receive queue due to low socket
>>> buffer
>>>
>>> I've increased to socket buffer size, backlog queue size and memlock
>>> value was set to unlimited for ats user. How can I fix it completely?
>>>
>>> Best Regards,
>>> Ergin Ozekes
>>>
>>
>>
>
Re: packets collapsed in receive queue due to low socket buffer
Posted by Ergin Ozekes <er...@gmail.com>.
Hi all;
In according to my investigation on the ats read
performance,NetHandler::mainNetEvent is the top function. On my system
lots of buffer over run is exist. In my system net input interface has 250
Mbps traffic but on the client output only 125 Mbps is exist. Half of the
downloaded content is returning to client.
Does anyone knows the reason of this case?
And how can I fixe it?
This is the "perf top" output:
4.82% libpcre.so.3.13.1 [.] match
3.27% [kernel] [k] native_write_msr_safe
2.09% [kernel] [k] ipt_do_table
1.60% [kernel] [k] __schedule
1.57% [kernel] [k] menu_select
1.43% libc-2.13.so [.] 0x00000000000e92b0
1.35% [kernel] [k] find_busiest_group
1.16% traffic_server [.] EThread::execute()
1.12% [kernel] [k] copy_user_generic_string
1.10% [kernel] [k] nf_iterate
1.04% [kernel] [k] _raw_spin_lock_irqsave
1.03% [kernel] [k] int_sqrt
*1.02% traffic_server [.] NetHandler::mainNetEvent(int,
Event*)*
0.96% [bnx2x] [k] bnx2x_rx_int
0.93% [kernel] [k] _raw_spin_lock
0.90% [kernel] [k] htable_selective_cleanup
0.86% [kernel] [k] cpuidle_enter_state
0.83% [kernel] [k] enqueue_task_fair
0.83% [kernel] [k] tcp_packet
0.76% [kernel] [k] apic_timer_interrupt
0.76% [kernel] [k] timerqueue_add
0.76% [kernel] [k] idle_cpu
0.71% [bnx2x] [k] bnx2x_start_xmit
0.67% [kernel] [k] rb_erase
*0.64% traffic_server [.] read_from_net(NetHandler*,
UnixNetVConnection*, EThread*) *
On Thu, Oct 23, 2014 at 12:33 PM, Ergin Ozekes <er...@gmail.com>
wrote:
> Hi all;
>
> My current working network bandwidth is 350 Mbps.
> My netstat -st output prints out below value increasingly.
>
> 1555903887 packets collapsed in receive queue due to low socket buffer
> 1555908175 packets collapsed in receive queue due to low socket buffer
> 1555912925 packets collapsed in receive queue due to low socket buffer
> 1555920054 packets collapsed in receive queue due to low socket buffer
> 1555929162 packets collapsed in receive queue due to low socket buffer
> 1555938162 packets collapsed in receive queue due to low socket buffer
> 1555945682 packets collapsed in receive queue due to low socket buffer
> 1555951783 packets collapsed in receive queue due to low socket buffer
> 1555959318 packets collapsed in receive queue due to low socket buffer
> 1555962474 packets collapsed in receive queue due to low socket buffer
> 1555969574 packets collapsed in receive queue due to low socket buffer
>
> I've increased to socket buffer size, backlog queue size and memlock value
> was set to unlimited for ats user. How can I fix it completely?
> BTW, How can I increase read from socket(connection) performance of the
> ATS?
>
>
> Best Regards,
> Ergin Ozekes
>
> On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes <er...@gmail.com>
> wrote:
>
>> Hi all;
>>
>> My current working network bandwidth is 350 Mbps.
>> My netstat -st output prints out below value increasingly.
>>
>> 1555903887 packets collapsed in receive queue due to low socket buffer
>> 1555908175 packets collapsed in receive queue due to low socket buffer
>> 1555912925 packets collapsed in receive queue due to low socket buffer
>> 1555920054 packets collapsed in receive queue due to low socket buffer
>> 1555929162 packets collapsed in receive queue due to low socket buffer
>> 1555938162 packets collapsed in receive queue due to low socket buffer
>> 1555945682 packets collapsed in receive queue due to low socket buffer
>> 1555951783 packets collapsed in receive queue due to low socket buffer
>> 1555959318 packets collapsed in receive queue due to low socket buffer
>> 1555962474 packets collapsed in receive queue due to low socket buffer
>> 1555969574 packets collapsed in receive queue due to low socket buffer
>>
>> I've increased to socket buffer size, backlog queue size and memlock
>> value was set to unlimited for ats user. How can I fix it completely?
>>
>> Best Regards,
>> Ergin Ozekes
>>
>
>