Running dtrace inside a jail

2020-04-23
6 min read

In FreeBSD 12.2-RELEASE seems that dtrace is working inside a jail, trying this one liner on jail, seems to work but not quite.

dtraceall~load~="YES" must be added to /boot/loader.conf

root@yacy00:~ # dtrace -n 'syscall::open*:entry { printf("%s %s", execname, copyinstr(arg0)); }'
dtrace: description 'syscall::open*:entry ' matched 4 probes

dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
CPU     ID                    FUNCTION:NAME
  3  72450                       open:entry cron /etc/cron.d
  2  72450                       open:entry cron /etc/cron.d
  0  72450                       open:entry cron /etc/cron.d
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
  0  72450                       open:entry java /usr/home/yacy00/yacy/DATA/SURROGATES/in
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
  0  72450                       open:entry java /usr/home/yacy00/yacy/DATA/SURROGATES/in
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
  1  72450                       open:entry java /usr/home/yacy00/yacy/DATA/INDEX/freeworld/SEGMENTS/default
  1  72450                       open:entry java /usr/home/yacy00/yacy/DATA/INDEX/freeworld/SEGMENTS/default
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
  2  72450                       open:entry java /etc/hosts
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
  0  72450                       open:entry java /etc/hosts
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
  1  72450                       open:entry java /usr/home/yacy00/yacy/DATA/SURROGATES/in
dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12
^C

This error was not expected.

dtrace: error on enabled probe ID 1 (ID 73420: syscall:freebsd:openat:entry): invalid address (0xffffffffffffff9c) in action #2 at DIF offset 12

A quick search on the FreeBSD forums tells us about the error and a workaround. User Bojan Petrovic explains his findings:

After bisecting the kernel, I think the issue is related to commit r336876 ("Use SMAP on amd64") , mentioned in PR 230993 and PR 230460.

It seems to be resolved in the CURRENT and 12-STABLE, possibly in r342053 ("fix userspace access on boxes with SMAP") (didn't test).

If anyone is running a 12.0-RELEASE, the issue can be circumvented by disabling SMAP: add hw.cpu~stdextdisable~=0x00100000 to /boot/loader.conf

Problems

Dtrace works in a jail but it can see programs from jail0 as well, in this example pid 1994 is on jail 0 and pid 1942 is inside a jail

root@yacy00:~ # cat ex.d 
syscall:::
/probefunc!="read" && execname=="java"/
{
    printf("probefunc:%s, pid:%d, execname:%s\n",probefunc, pid, execname);
}


root@yacy00:~ # 
root@yacy00:~ # top
root@yacy00:~ # dtrace -qs ex.d
probefunc:nanosleep, pid:1994, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:nanosleep, pid:1942, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:open, pid:1994, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:open, pid:1994, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:fstatfs, pid:1994, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:fstatfs, pid:1994, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:getdirentries, pid:1994, execname:cron
probefunc:open, pid:1942, execname:cron
probefunc:open, pid:1942, execname:cron
probefunc:fstatfs, pid:1942, execname:cron
probefunc:getdirentries, pid:1994, execname:cron
probefunc:fstatfs, pid:1942, execname:cron
probefunc:getdirentries, pid:1942, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:getdirentries, pid:1994, execname:cron
probefunc:getdirentries, pid:1994, execname:cron
probefunc:getdirentries, pid:1942, execname:cron
probefunc:close, pid:1994, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:close, pid:1994, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:getdirentries, pid:1942, execname:cron
probefunc:fstatat, pid:1994, execname:cron
probefunc:nanosleep, pid:1994, execname:cron
probefunc:getdirentries, pid:1942, execname:cron
probefunc:close, pid:1942, execname:cron
probefunc:close, pid:1942, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:fstatat, pid:1942, execname:cron
probefunc:nanosleep, pid:1942, execname:cron
^C
probefunc:nanosleep, pid:1942, execname:cron
probefunc:nanosleep, pid:1994, execname:cron
probefunc:fstatat, pid:1942, execname:cron

References

https://forums.freebsd.org/threads/dtrace-shows-invalid-addres-errors-on-12-0-release-on-thinkpad-t480.69193/

https://forums.freebsd.org/threads/dtrace-openjdk12.71470/