You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "David Robinson (JIRA)" <ji...@apache.org> on 2011/05/11 01:41:47 UTC
[jira] [Created] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Traffic server has a hard limit of 512 gigabytes per RAW disk partition
-----------------------------------------------------------------------
Key: TS-773
URL: https://issues.apache.org/jira/browse/TS-773
Project: Traffic Server
Issue Type: Bug
Components: Cache
Affects Versions: 2.1.8
Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
12 1.5TB harddrives for cache disks.
Reporter: David Robinson
Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
storage.config is setup like this,
/dev/sda
/dev/sdb
/dev/sde
/dev/sdf
/dev/sdh
/dev/sdi
/dev/sdj
/dev/sdk
/dev/sdl
/dev/sdm
/dev/sdn
/dev/sdo
fdisk -l /dev/sdo
Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033852#comment-13033852 ]
Leif Hedstrom commented on TS-773:
----------------------------------
Yep, that looks better, there are two warnings at startup, but I think that's expected:
[May 15 18:45:44.543] Server {140432254875392} NOTE: reading directory '/dev/sdi 368640:244181087'
[May 15 18:45:44.595] Server {140432254875392} NOTE: using directory A for '/dev/sdh 368640:244181087'
[May 15 18:45:44.603] Server {140432254875392} NOTE: using directory A for '/dev/sdi 368640:244181087'
[May 15 18:47:55.534] Server {140432248559360} WARNING: bad footer in cache directory for '/dev/sdh 368640:244181087', clearing
[May 15 18:47:55.534] Server {140432248559360} NOTE: clearing cache directory '/dev/sdh 368640:244181087'
[May 15 18:47:55.542] Server {140432247506688} WARNING: bad footer in cache directory for '/dev/sdi 368640:244181087', clearing
[May 15 18:47:55.542] Server {140432247506688} NOTE: clearing cache directory '/dev/sdi 368640:244181087'
It does take a while to initialize and startup with 4TB of USB storage ;). After the startup has finished, things seems to work fine, and the cache size report seems good:
proxy.process.cache.bytes_total=3995674296320
proxy.node.cache.bytes_total=3995674345472
proxy.node.cache.bytes_total_mb=3810572
proxy.node.cache.bytes_free=3995674345472
proxy.node.cache.bytes_free_mb=3810572
Thanks John!
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
> Assignee: John Plevyak
> Fix For: 2.1.9
>
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom updated TS-773:
-----------------------------
Fix Version/s: 2.1.9
Assignee: John Plevyak
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
> Assignee: John Plevyak
> Fix For: 2.1.9
>
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "John Plevyak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032061#comment-13032061 ]
John Plevyak commented on TS-773:
---------------------------------
I checked in what I hope is the fix. The cplist code needs to be cleaned up
as it is not clear which counts are counting bytes/store blocks/disk volume blocks.
WARNING: this fix required changing the disk structure which will result in a cache WIPE!
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
> Assignee: John Plevyak
> Fix For: 2.1.9
>
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "David Robinson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031478#comment-13031478 ]
David Robinson commented on TS-773:
-----------------------------------
Debug log of all 12 disks
http://pastebin.com/S343UjCV
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032069#comment-13032069 ]
Leif Hedstrom commented on TS-773:
----------------------------------
John: Do you foresee more fixes for this bug before v2.1.9, or does this "complete" this bug?
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
> Assignee: John Plevyak
> Fix For: 2.1.9
>
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "John Plevyak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032119#comment-13032119 ]
John Plevyak commented on TS-773:
---------------------------------
Unfortunately, the hosting code is pretty complicated and poorly written.
I have tried to patch it, but it really should have a more comprehensive
audit. If this patch fixes the immediate problem (which it seems to have, but
I'd like independent confirmation) we can put off the audit till after 3.0.
So I don't foresee more changes if we get the fix verified.... but I wouldn't be
surprised if there was still a problem under some circumstances... until a full
audit.
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
> Assignee: John Plevyak
> Fix For: 2.1.9
>
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "John Plevyak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033816#comment-13033816 ]
John Plevyak commented on TS-773:
---------------------------------
Well, I did predict failure :) In this case the problem was that the directory can now be more than
2GB in size, which exceeds an 'int'. The resulting patch touched lots of the system because it also means
we can now read and write > 2GB in a single go. I have submitted a patch and tested in a faked disk over 2TB
(I had Store.cc lie about the size of the disk). Give it a go.
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
> Assignee: John Plevyak
> Fix For: 2.1.9
>
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033705#comment-13033705 ]
Leif Hedstrom commented on TS-773:
----------------------------------
I've tested this with 2x 2TB disks, that I mounted using a USB adaptor on my host (I'm assuming that shouldn't matter). In any case, trying it, I get a segfault:
[May 15 09:25:58.284] Server {140021139494656} NOTE: clearing cache directory '/dev/sdh 368640:244181087'
NOTE: Traffic Server received Sig 11: Segmentation fault
./bin/traffic_server - STACK TRACE:
Segmentation fault
(gdb) bt
#0 __memset_sse2 () at ../sysdeps/x86_64/memset.S:65
#1 0x00000000006488f8 in vol_clear_init (d=0x7fffe40042c0) at Cache.cc:982
#2 0x000000000064b84e in Vol::clear_dir (this=<value optimized out>) at Cache.cc:1014
#3 0x000000000064f928 in Vol::handle_header_read (this=0x7fffe40042c0, event=<value optimized out>,
data=<value optimized out>) at Cache.cc:1487
#4 0x0000000000652a15 in handleEvent (this=<value optimized out>, event=<value optimized out>,
data=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:146
#5 AIOCallbackInternal::io_complete (this=<value optimized out>, event=<value optimized out>,
data=<value optimized out>) at ../../iocore/aio/P_AIO.h:80
#6 0x00000000006dd7ee in handleEvent (this=0x7ffff711a010, e=0xf9a070, calling_code=1) at I_Continuation.h:146
#7 EThread::process_event (this=0x7ffff711a010, e=0xf9a070, calling_code=1) at UnixEThread.cc:140
#8 0x00000000006de2d3 in EThread::execute (this=0x7ffff711a010) at UnixEThread.cc:232
#9 0x00000000006dd42a in spawn_thread_internal (a=0xf83f90) at Thread.cc:85
#10 0x0000003c5de06ccb in start_thread (arg=0x7ffff670f700) at pthread_create.c:301
#11 0x0000003c5d2e0c2d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
> Assignee: John Plevyak
> Fix For: 2.1.9
>
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (TS-773) Traffic server has a hard limit of 512
gigabytes per RAW disk partition
Posted by "John Plevyak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Plevyak closed TS-773.
---------------------------
Resolution: Fixed
This seems to be fixed now. Tested by jplevyak and zwoop.
> Traffic server has a hard limit of 512 gigabytes per RAW disk partition
> -----------------------------------------------------------------------
>
> Key: TS-773
> URL: https://issues.apache.org/jira/browse/TS-773
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 2.1.8
> Environment: Debian Lenny 5.0.8 2.6.34.7 x86_64
> 12 1.5TB harddrives for cache disks.
> Reporter: David Robinson
> Assignee: John Plevyak
> Fix For: 2.1.9
>
>
> Using 1.5TB harddrives as cache disks results in ATS only using 512GBs of the disk. The disks are configured in RAW mode with no partition information.
> storage.config is setup like this,
> /dev/sda
> /dev/sdb
> /dev/sde
> /dev/sdf
> /dev/sdh
> /dev/sdi
> /dev/sdj
> /dev/sdk
> /dev/sdl
> /dev/sdm
> /dev/sdn
> /dev/sdo
> fdisk -l /dev/sdo
> Disk /dev/sdo: 1500.3 GB, 1500301910016 bytes
> 255 heads, 63 sectors/track, 182401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
> Partitioning a disk into 3 512G partition and adding then to storage.config will make ATS use the entire 1.5TBs of space.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira