It seems that trim support has been ported to illumos but is not clear
IIRC recommendations for a SLOG were that it should hold several (4?)
bursts of backend pool TXG size
<jimklimov> e.g. if you have 4 HDDs
that can write 150MB/s sequentially, and a raidz1, you peak at 450MB/s.
With 5-second TXGs an ideal burst might be over 2Gb, so your SLOG might
need at least 10Gb (if I remember the 4x correctly)
<jimklimov> typically your writes and bursts on moderately sized HDD-backed pools
would be smaller, so battery-backed RAM disks had a good business case
to host ZIL devices even at 4-8Gb sizes; I think if the logged writes
did not yet spill to the pool, the overflow would go to it directly (as
if when you have no SLOG device) so it is a performance drop but should
not be a showstopper (as in a system crash etc) \<jimklimov\> I saw
argumentation also in favor of dedicating SLOGs in all-flash arrays too,
to help wear-leveling: the cache devices write linear LBAs so can end up
reprogramming whole pages with little waste, and the pool devices would
get large write bursts at TXG flushes and also use the transistors
efficiently
<jimklimov> and the rest of the time those pool SSDs would
serve reads
<neirac> jimklimov thank you very much!.
<neirac> is trim setup in illumos by default ?
<jimklimov> not sure; I think support for the commands was eventually implemented (ported from Nexenta
maybe?) but no idea if it is enabled
<neirac> it seems we have trim <https://www.illumos.org/rb/r/2278/>
<neirac> but according to this is not ported <https://openzfs.org/wiki/Features#TRIM_Support>
<neirac> zpool trim seems to be the command, I\'ll check
Testing without zil and cache
cneira@cl-west-00:~$ dd if=/dev/zero of=~/temp.out bs=1024000 count=10240
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 2.587772 secs (3.77GB/sec)
cneira@cl-west-00:~$ dd if=~/temp.out of=/dev/null bs=1024000
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 1.311424 secs (7.45GB/sec)
cneira@cl-west-00:~$ ls
temp.out
cneira@cl-west-00:~$ rm temp.out
After adding
cneira@cl-west-00:~$ dd if=/dev/zero of=~/temp.out bs=1024000 count=10240
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 2.088420 secs (4.68GB/sec)
cneira@cl-west-00:~$ dd if=~/temp.out of=/dev/null bs=1024000
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 1.309615 secs (7.46GB/sec)
FreeBSD results
cneira@cl-west-00:~ $ dd if=/dev/zero of=~/temp.out bs=1024000 count=10240
dd: ~/temp.out: No such file or directory
cneira@cl-west-00:~ $ dd if=/dev/zero of=./temp.out bs=1024000 count=10240
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 2.090663 secs (5015519137 bytes/sec)
cneira@cl-west-00:~ $ dd if=./temp.out of=/dev/null bs=1024000 count=10240
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 1.227734 secs (8540741773 bytes/sec)
After log and cache
cneira@cl-west-00:~ $ dd if=/dev/zero of=./temp.out bs=1024000 count=10240
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 1.972901 secs (5314894845 bytes/sec)
cneira@cl-west-00:~ $ dd if=./temp.out of=/dev/null bs=1024000 count=10240
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 1.174418 secs (8928476617 bytes/sec)
Adding log and cache in FreeBSD
I'm using 16G as my machine has 32GB an slog could only hold half the host memory, so the rest of space will be used as cache.
root@cl-west-00:/home/cneira # gpart destroy -F ada1
root@cl-west-00:/home/cneira # gpart create -s gpt ada1
root@cl-west-00:/home/cneira # gpart add -t freebsd-zfs -b 2048 -a 4k -l log0 -s 16G ada1
ada1p1 added
root@cl-west-00:/home/cneira # gpart add -t freebsd-zfs -a 4k -l cache0 ada1
ada1p2 added
root@cl-west-00:/home/cneira # zpool add zroot log gpt/log0
invalid vdev specification
use '-f' to override the following errors:
/dev/gpt/log0 is part of potentially active pool 'zroot'
root@cl-west-00:/home/cneira # zpool add -f zroot log gpt/log0
root@cl-west-00:/home/cneira # zpool add zroot cache gpt/cache0
root@cl-west-00:/home/cneira # zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
ada2p3 ONLINE 0 0 0
logs
gpt/log0 ONLINE 0 0 0
cache
gpt/cache0 ONLINE 0 0 0
errors: No known data errors
root@cl-west-00:/home/cneira #
https://wp.strahlert.net/wordpress/zfs-2/expanding-zpool/
Create ZIL and cache slices in illumos
To create 2 slices for log and cache we need to use format if partitions exists just erase them selecting 0 as size.
format
partition> 1
Part Tag Flag First Sector Size Last Sector
1 FreeBSD ZFS wm 33556480 133.05GB 312581767
Enter partition id tag[FreeBSD ZFS]:
Enter partition permission flags[wm]:
Enter new starting Sector[33556480]: 0
`0' is out of range.
Enter new starting Sector[33556480]:
Enter partition size[279025288b, 312581767e, 136242mb, 133gb, 0tb]: 0
partition> print
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 unassigned wm 0 0 0
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> l
Ready to label disk, continue? y
no reserved partition found
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 unassigned wm 0 0 0
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> 0
Part Tag Flag First Sector Size Last Sector
0 unassigned wm 0 0 0
Enter partition id tag[usr]: log
`log' not expected.
Enter partition id tag[usr]: slog
`slog' not expected.
Enter partition id tag[usr]:
Enter partition permission flags[wm]:
Enter new starting Sector[40]: 0
`0' is out of range.
Enter new starting Sector[40]:
Enter partition size[0b, 39e, 0mb, 0gb, 0tb]: 0
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 unassigned wm 0 0 0
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> print
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 unassigned wm 0 0 0
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> 0
Part Tag Flag First Sector Size Last Sector
0 unassigned wm 0 0 0
Enter partition id tag[usr]:
Enter partition permission flags[wm]:
Enter new starting Sector[40]:
Enter partition size[0b, 39e, 0mb, 0gb, 0tb]: 16gb
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> l
Ready to label disk, continue? y
no reserved partition found
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> 1
Part Tag Flag First Sector Size Last Sector
1 unassigned wm 0 0 0
Enter partition id tag[usr]:
Enter partition permission flags[wm]:
Enter new starting Sector[33554472]:
Enter partition size[0b, 33554471e, 0mb, 0gb, 0tb]:
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> 1
Part Tag Flag First Sector Size Last Sector
1 unassigned wm 0 0 0
Enter partition id tag[usr]: cache
`cache' not expected.
Enter partition id tag[usr]:
Enter partition permission flags[wm]:
Enter new starting Sector[33554472]:
Enter partition size[0b, 33554471e, 0mb, 0gb, 0tb]: 133gb
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 usr wm 33554472 133.00GB 312475687
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> 0
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
Enter partition id tag[usr]: unassigned
Enter partition permission flags[wm]:
Enter new starting Sector[40]:
Enter partition size[33554432b, 33554471e, 16384mb, 16gb, 0tb]:
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 usr wm 33554472 133.00GB 312475687
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> l
Ready to label disk, continue? y
no reserved partition found
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 usr wm 33554472 133.00GB 312475687
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> ^[^R
^R
partition> 0
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
Enter partition id tag[usr]: unassigned
Enter partition permission flags[wm]:
Enter new starting Sector[40]:
Enter partition size[33554432b, 33554471e, 16384mb, 16gb, 0tb]: 16gb
partition> p
Current partition table (unnamed):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 usr wm 33554472 133.00GB 312475687
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> l
Ready to label disk, continue? y
no reserved partition found
partition> l
Ready to label disk, continue? y
no reserved partition found
partition> q
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
fdisk - run the fdisk program
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
cneira@cl-west-00:~$ sudo format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c2t0d0 <ATA-ST2000NM0055 03X3795 LENOVO-LJ83-1.82TB>
/pci@0,0/pci17aa,30c1@17/disk@0,0
1. c2t2d0 <INTEL-SSDSA2CW160G3-4PC10362-149.05GB>
/pci@0,0/pci17aa,30c1@17/disk@2,0
2. c2t4d0 <WDC-WD20PURZ-85GU6Y0-80.00A80-1.82TB>
/pci@0,0/pci17aa,30c1@17/disk@4,0
Specify disk (enter its number): 1
selecting c2t2d0
[disk formatted]
Error: can't open disk '/dev/rdsk/c2t2d0p0'.
Note: capacity in disk label is smaller than the real disk capacity.
Select <partition> <expand> to adjust the label capacity.
No Solaris fdisk partition found.
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
fdisk - run the fdisk program
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> partition
PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
expand - expand label to use whole disk
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
!<cmd> - execute <cmd>, then return
quit
partition> p
Current partition table (original):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 usr wm 33554472 133.00GB 312475687
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> quit
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
fdisk - run the fdisk program
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> quit
cneira@cl-west-00:~$ sudo format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c2t0d0 <ATA-ST2000NM0055 03X3795 LENOVO-LJ83-1.82TB>
/pci@0,0/pci17aa,30c1@17/disk@0,0
1. c2t2d0 <INTEL-SSDSA2CW160G3-4PC10362-149.05GB>
/pci@0,0/pci17aa,30c1@17/disk@2,0
2. c2t4d0 <WDC-WD20PURZ-85GU6Y0-80.00A80-1.82TB>
/pci@0,0/pci17aa,30c1@17/disk@4,0
Specify disk (enter its number): 1
selecting c2t2d0
[disk formatted]
Error: can't open disk '/dev/rdsk/c2t2d0p0'.
Note: capacity in disk label is smaller than the real disk capacity.
Select <partition> <expand> to adjust the label capacity.
No Solaris fdisk partition found.
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
fdisk - run the fdisk program
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> partition
PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
expand - expand label to use whole disk
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
!<cmd> - execute <cmd>, then return
quit
partition> p
Current partition table (original):
Total disk sectors available: 312565344 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 40 16.00GB 33554471
1 usr wm 33554472 133.00GB 312475687
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
partition> q
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
fdisk - run the fdisk program
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> quit
cneira@cl-west-00:~$ sudo zpool add rpool log c2t2d0s0
cneira@cl-west-00:~$ sudo zpool add rpool cache c2t2d0s1
cneira@cl-west-00:~$ zpool status
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
logs
c2t2d0s0 ONLINE 0 0 0
cache
c2t2d0s1 ONLINE 0 0 0
errors: No known data errors
References
https://dan.langille.org/2016/03/06/adding-slog-to-a-zpool/ https://ownlinuxnotes.wordpress.com/2013/11/01/create-or-delete-partition-in-solaris/