Edgar Holddisk Benchmarks Using Linux Software Raid MD and different Filesystems.
First, all 14 Seagate 600GB SAS (15krpm) are in pass through mode on the 2 LSI-9211–8i PCIe cards, 6 on the first card and 8 on the second card due to the unfortunate fact that the system disks are on the same card (OS is installed in a raid1/mirror array constructed using the LSI BIOS utilities)). I would have preferred to have each of the 2 system disks on their own HBA and install the OS on a Linux SW (mdadm) raid1 volume and completely bypass the BIOS LSI raid stuff.
Note that the array creation in raid10 mode is with the near layout (the default) so it is probably worth it in this case to stagger the disks on each card, so that the mirror chunk is not on the same controller.
In a near=2 layout the data blocks are distributed like this for a 6 disk array: (the columns are the physical disk device)
----------------------- A1 A1 A2 A2 A3 A3 A4 A4 A5 A5 A6 A6 A7 A7 A8 A8 A9 A9 A10 A10 A11 A11 A12 A12 ... ... ... ... ... ...
For a far layout, the disks are split in f sections and each chunk is repeated with an offset. With f=2 and 6 disks:
------------------------ A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 A15 A16 A17 A18 ... ... ... ... ... ... A6 A1 A2 A3 A4 A5 A12 A7 A8 A9 A10 A11 A18 A13 A14 A15 A16 A17 ... ... ... ... ... ...
Near n2 layout
edgar:~# sfdisk -l /dev/sdb
and create a partition layout (md only works with disk partitions, not whole disk block device).
edgar:~# for part in c d e f g h i j k l m n o; do sfdisk -d /dev/sdb | sfdisk /dev/sd$part; done edgar:~# for part in b c d e f g h i j k l m n o; do sfdisk --change-id /dev/sd$part 1 fd; done edgar:~# mdadm --create --bitmap=internal --level=raid10 --raid-devices=14 \ /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 \ /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1 edgar:~# mdadm --examine --scan >> /etc/mdadm/mdadm.conf edgar:~# dpkg-reconfigure mdadm
- edgar:~# mkfs.xfs /dev/md0
edgar:~# mkdir /holddisk edgar:~# mount -t xfs /dev/md0 /holddisk edgar:~# chown amanda.disk /holddisk edgar:~# xfs_info /holddisk meta-data=/dev/md0 isize=256 agcount=32, agsize=32050048 blks = sectsz=512 attr=2 data = bsize=4096 blocks=1025601024, imaxpct=5 = sunit=128 swidth=896 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=500784, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 edgar:~# chown amanda.disk /holddisk edgar:~# bonnie++ -d /holddisk -s 320G -u amanda:disk -n 0 -f -b
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Version 1.96 | Sequential Output | Sequential Input | | | Sequential Create | Random Create | |-----------------+----------------------------------------------+------------------------------|Random |----------+-----------------------------------------+-----------------------------------------| | | Size | Per Char | Block | Rewrite | Per Char | Block | Seeks | Num | Create | Read | Delete | Create | Read | Delete | | | | | | | | | | Files | | | | | | | |-----------------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | K/ | % | K/sec | % | K/sec | % | K/ | % | K/sec | % | /sec | % | | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | | | sec | CPU | | CPU | | CPU | sec | CPU | | CPU | | CPU | | | CPU | | CPU | | CPU | | CPU | | CPU | | CPU | |-----------------+------+------+---------+------+--------+------+------+------+---------+------+-------+------+----------+------+------+------+------+------+------+------+------+------+------+------+------| | | 320G | | | 1091386 | 81 | 607099 | 52 | | | 1198788 | 46 | 373.2 | 23 | | | | | | | | | | | | | | |edgar |---------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | Latency | | 493ms | 287ms | | 152ms | 100ms | Latency | | | | | | | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- mkfs.xfs -f -d su=64k,sw=14 -l version=2,su=64k /dev/md0
edgar:~# xfs_info /holddisk meta-data=/dev/md0 isize=256 agcount=32, agsize=32050032 blk = sectsz=512 attr=2 data = bsize=4096 blocks=1025601024, imaxpct=5 = sunit=16 swidth=224 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=500784, version=2 = sectsz=512 sunit=16 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Version 1.96 | Sequential Output | Sequential Input | | | Sequential Create | Random Create | |-----------------+----------------------------------------------+------------------------------|Random |----------+-----------------------------------------+-----------------------------------------| | | Size | Per Char | Block | Rewrite | Per Char | Block | Seeks | Num | Create | Read | Delete | Create | Read | Delete | | | | | | | | | | Files | | | | | | | |-----------------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | K/ | % | K/sec | % | K/sec | % | K/ | % | K/sec | % | /sec | % | | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | | | sec | CPU | | CPU | | CPU | sec | CPU | | CPU | | CPU | | | CPU | | CPU | | CPU | | CPU | | CPU | | CPU | |-----------------+------+------+---------+------+--------+------+------+------+---------+------+-------+------+----------+------+------+------+------+------+------+------+------+------+------+------+------| | | 320G | | | 1099665 | 82 | 608266 | 52 | | | 1190683 | 45 | 357.6 | 22 | | | | | | | | | | | | | | |edgar |---------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | Latency | | 1289ms | 319ms | | 18831us | 121ms | Latency | | | | | | | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- mkfs.xfs -f -d su=64k,sw=7 -l version=2,su=64k /dev/md0
mkfs.xfs: Specified data stripe unit 128 is not the same as the volume stripe unit 1024 mkfs.xfs: Specified data stripe width 896 is not the same as the volume stripe width 7168 meta-data=/dev/md0 isize=256 agcount=32, agsize=32050016 blks = sectsz=512 attr=2, projid32bit=0 data = bsize=4096 blocks=1025600512, imaxpct=5 = sunit=16 swidth=112 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal log bsize=4096 blocks=500784, version=2 = sectsz=512 sunit=16 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Version 1.96 | Sequential Output | Sequential Input | | | Sequential Create | Random Create | |-----------------+----------------------------------------------+------------------------------|Random |----------+-----------------------------------------+-----------------------------------------| | | Size | Per Char | Block | Rewrite | Per Char | Block | Seeks | Num | Create | Read | Delete | Create | Read | Delete | | | | | | | | | | Files | | | | | | | |-----------------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | K/ | % | K/sec | % | K/sec | % | K/ | % | K/sec | % | /sec | % | | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | | | sec | CPU | | CPU | | CPU | sec | CPU | | CPU | | CPU | | | CPU | | CPU | | CPU | | CPU | | CPU | | CPU | |-----------------+------+------+---------+------+--------+------+------+------+---------+------+-------+------+----------+------+------+------+------+------+------+------+------+------+------+------+------| | | 320G | | | 1097755 | 82 | 609745 | 52 | | | 1191963 | 45 | 362.4 | 22 | | | | | | | | | | | | | | |edgar |---------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | Latency | | 1193ms | 247ms | | 45000us | 111ms | Latency | | | | | | | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- mkfs.ext4 /dev/md0
mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=128 blocks, Stripe width=896 blocks 256401408 inodes, 1025601024 blocks 51280051 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 31299 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848, 512000000, 550731776, 644972544 Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 21 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Version 1.96 | Sequential Output | Sequential Input | | | Sequential Create | Random Create | |-----------------+---------------------------------------------+------------------------------|Random |----------+-----------------------------------------+-----------------------------------------| | | Size | Per Char | Block | Rewrite | Per Char | Block | Seeks | Num | Create | Read | Delete | Create | Read | Delete | | | | | | | | | | Files | | | | | | | |-----------------+-------------+---------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | K/ | % | K/sec | % | K/sec | % | K/ | % | K/sec | % | /sec | % | | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | | | sec | CPU | | CPU | | CPU | sec | CPU | | CPU | | CPU | | | CPU | | CPU | | CPU | | CPU | | CPU | | CPU | |-----------------+------+------+--------+------+--------+------+------+------+---------+------+-------+------+----------+------+------+------+------+------+------+------+------+------+------+------+------| | | 320G | | | 923375 | 80 | 568904 | 57 | | | 1183264 | 43 | 290.0 | 35 | | | | | | | | | | | | | | |edgar |---------+-------------+---------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | Latency | | 1283ms | 255ms | | 188ms | 108ms | Latency | | | | | | | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Far f2 layout
Zap the previous ‘near’ layout and start over again:
# Create one partition spanning the entire block device. # sfdisk -l /dev/sdb # Duplicate it throughout the devices that will be used for the raid device. # for part in c d e f g h i j k l m n o; do sfdisk -d /dev/sdb | sfdisk /dev/sd$part; done # Change the partition id to '0xfd' or 'Linux raid autodetect'. # for part in b c d e f g h i j k l m n o; do sfdisk --change-id /dev/sd$part 1 fd; done # For good measure, zap any previous md raidness footprints: # for part in b1 c1 d1 e1 f1 g1 h1 i1 j1 k1 l1 m1 n1 o1; do mdadm --zero-superblock /dev/sd$part; done # Create the raid10 volume using '--layout=n2' for 'near' layout (default if # not specified) or '--layout=f2' for 'far' layout. # mdadm --create --bitmap=internal --level=raid10 --layout=f2 --raid-devices=14 /dev/md0 \ /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 \ /dev/sdk1 /dev/sdl1 /dev/sdm1 /dev/sdn1 /dev/sdo1
- mkfs.xfs /dev/md0
Use XFS default values:
edgar:~# xfs_info /holddisk
meta-data=/dev/md0 isize=256 agcount=32, agsize=32050048 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=1025601024, imaxpct=5
= sunit=128 swidth=1792 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=500784, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
bonnie++ -d /holddisk -s 320G -u amanda:disk -n 0 -f -b +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Version 1.96 | Sequential Output | Sequential Input | | | Sequential Create | Random Create | |-----------------+----------------------------------------------+------------------------------|Random |----------+-----------------------------------------+-----------------------------------------| | | Size | Per Char | Block | Rewrite | Per Char | Block | Seeks | Num | Create | Read | Delete | Create | Read | Delete | | | | | | | | | | Files | | | | | | | |-----------------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | K/ | % | K/sec | % | K/sec | % | K/ | % | K/sec | % | /sec | % | | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | | | sec | CPU | | CPU | | CPU | sec | CPU | | CPU | | CPU | | | CPU | | CPU | | CPU | | CPU | | CPU | | CPU | |-----------------+------+------+---------+------+--------+------+------+------+---------+------+-------+------+----------+------+------+------+------+------+------+------+------+------+------+------+------| | | 320G | | | 1043392 | 79 | 580224 | 51 | | | 2305452 | 94 | 331.6 | 21 | | | | | | | | | | | | | | |edgar |---------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | Latency | | 503ms | 214ms | | 176ms | 153ms | Latency | | | | | | | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- edgar:~# mkfs.xfs -f -d su=64k,sw=14 -l version=2,su=64k /dev/md0
mkfs.xfs: Specified data stripe unit 128 is not the same as the volume stripe un
mkfs.xfs: Specified data stripe width 1792 is not the same as the volume stripe
meta-data=/dev/md0 isize=256 agcount=32, agsize=32050032 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=1025601024, imaxpct=5
= sunit=16 swidth=224 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=500784, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
edgar:~# bonnie++ -d /holddisk -s 320G -u amanda:disk -n 0 -f -b
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Version 1.96 | Sequential Output | Sequential Input | | | Sequential Create | Random Create | |-----------------+----------------------------------------------+------------------------------|Random |----------+-----------------------------------------+-----------------------------------------| | | Size | Per Char | Block | Rewrite | Per Char | Block | Seeks | Num | Create | Read | Delete | Create | Read | Delete | | | | | | | | | | Files | | | | | | | |-----------------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | K/ | % | K/sec | % | K/sec | % | K/ | % | K/sec | % | /sec | % | | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | | | sec | CPU | | CPU | | CPU | sec | CPU | | CPU | | CPU | | | CPU | | CPU | | CPU | | CPU | | CPU | | CPU | |-----------------+------+------+---------+------+--------+------+------+------+---------+------+-------+------+----------+------+------+------+------+------+------+------+------+------+------+------+------| | | 320G | | | 1053321 | 79 | 581534 | 51 | | | 2320811 | 94 | 332.1 | 21 | | | | | | | | | | | | | | |edgar |---------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | Latency | | 918ms | 264ms | | 35130us | 126ms | Latency | | | | | | | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- mkfs.xfs -f -d su=64k,sw=7 -l version=2,su=64k /dev/md0
edgar:~# xfs_info /holddisk
meta-data=/dev/md0 isize=256 agcount=32, agsize=32050016 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=1025600512, imaxpct=5
= sunit=16 swidth=112 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=500784, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
edgar:~# bonnie++ -d /holddisk -s 320G -u amanda:disk -n 0 -f -b
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Version 1.96 | Sequential Output | Sequential Input | | | Sequential Create | Random Create | |-----------------+----------------------------------------------+------------------------------|Random |----------+-----------------------------------------+-----------------------------------------| | | Size | Per Char | Block | Rewrite | Per Char | Block | Seeks | Num | Create | Read | Delete | Create | Read | Delete | | | | | | | | | | Files | | | | | | | |-----------------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | K/ | % | K/sec | % | K/sec | % | K/ | % | K/sec | % | /sec | % | | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | /sec | % | | | sec | CPU | | CPU | | CPU | sec | CPU | | CPU | | CPU | | | CPU | | CPU | | CPU | | CPU | | CPU | | CPU | |-----------------+------+------+---------+------+--------+------+------+------+---------+------+-------+------+----------+------+------+------+------+------+------+------+------+------+------+------+------| | | 320G | | | 1048710 | 78 | 583268 | 51 | | | 2295685 | 93 | 328.0 | 21 | | | | | | | | | | | | | | |edgar |---------+-------------+----------------+---------------+-------------+----------------+--------------+----------+-------------+-------------+-------------+-------------+-------------+-------------| | | Latency | | 405ms | 200ms | | 161ms | 125ms | Latency | | | | | | | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+