I run a lot of hardware that’s well over a decade old. Up until last year I exclusively used an HP USDT as my daily driver. While it often works without issue, many “modern” applications are resource intensive. Often leading to cpu starvation or memory contention. In order to keep the desktop environment afloat in these situations I’ve been wrapping commands in cgroups using systemd to enforce memory limits on them. Even though I have had a modern laptop for the past year, I still enforce these same limits1. I figured I’d share my hacky convenient zsh aliases for performing such things.
This isn’t so much an actual post as it is a dump of some aliases.
Dispatching commands
I’m not a complete systemd fanboy, but one of the nice things is how process groups have a pair of abstractions layers, the slice and the unit, thrown on top of them.2 This lets you group commands into units, and units into slices. You can carve up a slice as a category; imposing a limit on the resource sums for all units in said slice. Then each unit can use individual resource limits. If a unit hits its individual limit, it will be throttled, start swapping, or be killed individually. But if the sum of all the units in a slice hit the slice limits, they will be penalized as a whole.
User slices follow the following format3
~ cat ~/.config/systemd/user/browsers.slice
[Unit]
Description=browsers
DefaultDependencies=no
Before=slices.target
Requires=system.slice
After=system.slice
[Slice]
MemoryHigh=10G
MemoryMax=11G
MemorySwapMax=1024M
I wrap it in a dispatching function, mainly so I can plumb in changes centrally.
function dispatch {
if [[ $USER == "root" ]]; then
command "$binary" "$@"
return $?
fi
declare args=(
--user
--same-dir
-p IOAccounting=yes
-p MemoryAccounting=yes
-p TasksAccounting=yes
)
while (($#)); do
case "$1" in
-c)
args+="-p"
args+="CPUWeight=$2"; shift 2
;;
-mm)
args+="-p"
args+="MemoryMax=$2"; shift 2
;;
-mh)
args+="-p"
args+="MemoryHigh=$2"; shift 2
;;
-s)
args+="-p"
args+="MemorySwapMax=$2"; shift 2
;;
--scope)
args+=--scope; shift
;;
--slice)
args+="--slice=$2"; shift 2
;;
--name)
name=$1; shift 2
;;
-P)
args+=-P; shift
;;
--binary)
[ -z "$name" ] || name=$2
binary="$2"; shift
;;
*)
break
esac
done
systemd-run $args "$@" 2> >(>&2 grep -vE 'Running.*as unit:')
}
Now, this works all well and good from a terminal. But a lot of processes call fork/execv and for those I drop a wrapper in a ~/bin directory that I’ve added to my path.
~ cd ~/bin
~ cat .dispatch/template
#!/usr/bin/env zsh
# redirecting source to dev null is important when you're parsing output
source ~/.zshrc >/dev/null 2>/dev/null
"${ZSH_ARGZERO##*/}" $@
~ ln -s .dispatch/template firefox
This is a must if you want to wrap commands from your launcher or some of your lsp evoked commands. 4 That’s basically it for the mechanics of the thing.
A little more nuance
While I’m on the topic of lsp/development environments; so many commands use node that I wrap those separately.
function node {
case "$1" in
*"typescript-language-server"*)
dispatch --name tsserver \
--scope \
--slice node \
-c 3000 \
-mh 3.5G \
-mm 3.6G \
-s 128M \
--binary /usr/bin/node "$@"
;;
*"pyright"*)
dispatch --name pyright \
--scope \
--slice pythonlsp.slice \
-c 100 \
-mh 2000M \
-mm 2048M \
-s 48M \
--binary /usr/bin/node "$@"
;;
*"tsc"*)
dispatch \
--name tsc \
--scope \
--slice node \
-c 135 \
-mh 2000M \
-mm 2048M \
-s 24M \
--binary /usr/bin/node "$@"
;;
*"cdktf"*)
dispatch \
--name cdktf \
--scope \
--slice node \
-c 35 \
-mh 3G \
-mm 3G \
-s 24M \
--binary /usr/bin/node "$@"
;;
*"eslint_d"* | *"core_d"*)
dispatch \
--name eslint_d \
--scope \
--slice node \
-c 200 \
-mh 4.3G \
-mm 4.5G \
-s 24M \
--binary /usr/bin/node "$@"
;;
*"eslint"*)
dispatch \
--name eslint \
--scope \
--slice node \
-c 300 \
-mh 2.5G \
-mm 2.6G \
-s 24M \
--binary /usr/bin/node "$@"
;;
*"prettier-eslint"*)
dispatch \
--name prettier-eslint \
--scope \
--slice node \
-c 300 \
-mh 500M \
-mm 512M \
-s 12M \
--binary /usr/bin/node "$@"
;;
*"prettier"*)
dispatch \
--name prettier \
--scope \
--slice node \
-c 300 \
-mh 500M \
-mm 512M \
-s 12M \
--binary /usr/bin/node "$@"
;;
*"diagnostic-languageserver"*)
dispatch \
--name diagnostic-languageserver \
--scope \
--slice node \
-c 90 \
-mh 1024M \
-mm 1024M \
-s 12M \
--binary /usr/bin/node "$@"
;;
*"npm"*)
dispatch \
--name npm \
--scope \
--slice node \
-c 35 \
-mh 1500M \
-mm 1572M\
-s 72M \
--binary /usr/bin/node "$@"
;;
*)
dispatch \
--name node-generic \
--scope \
--slice node \
-c 35 \
-mh 500M \
-mm 512M \
-s 12M \
--binary /usr/bin/node "$@"
;;
esac
}
I match python similarly 5
python () {
if [ -n "$VIRTUAL_ENV" ]; then
source $VIRTUAL_ENV/bin/activate
fi
case "$1" in
*"jedi"*)
dispatch --name jedi-language-server \
--scope \
--slice pythonlsp.slice \
-c 100 \
-mh 2000M \
-mm 2048M \
-s 48M \
--binary python "$@"
;;
*)
dispatch \
--name python-generic \
--scope \
--slice pythonlsp.slice \
-c 35 \
-mh 500M \
-mm 512M \
-s 12M \
--binary python "$@"
;;
esac
}
Introspection
You can check on the status of it with systemctl --user status browsers.slice. Example output is as follows. Notice how there are two scope units nested under the slice. The firefox and chrome pids are nested under their respective scope.
● browsers.slice - browsers
Loaded: loaded (/home/matt/.config/systemd/user/browsers.slice; static)
Drop-In: /home/matt/.config/systemd/user.control/browsers.slice.d
└─50-MemoryHigh.conf, 50-MemoryMax.conf
Active: active since Wed 2022-06-22 05:54:52 AKDT; 5 days ago
Until: Wed 2022-06-22 05:54:52 AKDT; 5 days ago
Tasks: 753
Memory: 3.5G (high: 10.1G max: 10.0G swap max: 1.0G available: 6.4G)
CPU: 10h 24min 23.110s
CGroup: /user.slice/user-10010.slice/user@10010.service/browsers.slice
├─run-rd666fcddb4584959b3e72ec7422f707f.scope
│ ├─2707939 /usr/lib/firefox/firefox
│ ├─2708074 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220609170544 -prefsLen 9513 -prefMapSize 266077 -appDir /usr/lib/fir>
│ ├─2708100 /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -prefsLen 9631 -prefMapSize 266077 -jsInitLen 277128 -parent>
│ ├─2708143 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -prefsLen 14640 -prefMapSize 266077 -jsInitLen 277128 -paren>
│ ├─3369479 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220609170544 -prefsLen 18146 -prefMapSize 266077 -appDir /usr/lib/fi>
│ ├─3374574 /usr/lib/firefox/firefox -contentproc -childID 161 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3473440 /usr/lib/firefox/firefox -contentproc -childID 168 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3585007 /usr/lib/firefox/plugin-container /home/matt/.mozilla/firefox/i3w1cbew.default/gmp-widevinecdm/4.10.2449.0 2707939 true gmp>
│ ├─3619513 /usr/lib/firefox/firefox -contentproc -childID 195 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3619611 /usr/lib/firefox/firefox -contentproc -childID 196 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3619778 /usr/lib/firefox/firefox -contentproc -childID 197 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3768700 /usr/lib/firefox/firefox -contentproc -childID 210 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3774713 /usr/lib/firefox/firefox -contentproc -childID 211 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3887801 /usr/lib/firefox/firefox -contentproc -childID 213 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3930854 /usr/lib/firefox/firefox -contentproc -childID 217 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3933460 /usr/lib/firefox/firefox -contentproc -childID 218 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3933865 /usr/lib/firefox/firefox -contentproc -childID 219 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ ├─3935821 /usr/lib/firefox/firefox -contentproc -childID 220 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
│ └─3936239 /usr/lib/firefox/firefox -contentproc -childID 221 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -par>
└─run-re6a64a3f61e9425280a6cc758d955648.scope
├─3927358 "/usr/lib/chromium/chromium --enable-features=UseOzonePlatform --ozone-platform=wayland
├─3927371 /usr/lib/chromium/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler "--database=/home>
├─3927373 /usr/lib/chromium/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler "--database=>
├─3927377 "/usr/lib/chromium/chromium --type=zygote --no-zygote-sandbox --enable-crashpad --crashpad-handler-pid=3927371 --enable-cra>
├─3927378 "/usr/lib/chromium/chromium --type=zygote --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
├─3927380 "/usr/lib/chromium/chromium --type=zygote --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
├─3927405 "/usr/lib/chromium/chromium --type=gpu-process --ozone-platform=wayland --enable-crashpad --crashpad-handler-pid=3927371 -->
├─3927406 "/usr/lib/chromium/chromium --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-t>
├─3927411 "/usr/lib/chromium/chromium --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-t>
├─3927482 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch >
├─3927483 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch >
└─3927558 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch >
Jun 22 05:54:52 matt-gen-laptop-p01 systemd[809]: Created slice browsers.
I use the brittle peek alias often
function peek {
[ -n "$1" ] || bail 1 "specify a slice"
fid=()
while read unit; do
fid+=r"$unit"
done < <(systemctl --user status "$1" | grep scope | cut -f 2- -d r)
systemctl --user status $fid
}
which in turn relies on bail, but that isn’t a hard requirement
function bail {
if [ -z "$1" ]; then
printf "no exit code returned\n"
return 1
elif [ "$1" -ne 0 ]; then
[[ -z "$2" ]] && printf "failed\n" || printf "%s\n" "$2"
fi
return "$1"
}
A variant of peek obcess
function obcess {
[ -n "$1" ] || bail 1 "specify a slice"
watch 'zsh -c "source /home/matt/.zshrc >/dev/null 2>/dev/null && peek '$1'"'
}
Peek in action.
~ peek browsers.slice
● run-rd666fcddb4584959b3e72ec7422f707f.scope - /usr/bin/firefox
Loaded: loaded (/run/user/10010/systemd/transient/run-rd666fcddb4584959b3e72ec7422f707f.scope; transient)
Transient: yes Active: active (running) since Thu 2022-06-23 21:13:14 AKDT; 3 days ago
Tasks: 620 (limit: 33435)
Memory: 2.6G (high: 6.3G max: 6.3G swap max: 12.0M available: 3.6G)
CPU: 8h 10min 52.728s
CGroup: /user.slice/user-10010.slice/user@10010.service/browsers.slice/run-rd666fcddb4584959b3e72ec7422f707f.scope
├─2707939 /usr/lib/firefox/firefox
├─2708074 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220609170544 -prefsLen 9513 -prefMapSize 266077 -appDir /usr/lib/firef>
├─2708100 /usr/lib/firefox/firefox -contentproc -childID 1 -isForBrowser -prefsLen 9631 -prefMapSize 266077 -jsInitLen 277128 -parentBu>
├─2708143 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -prefsLen 14640 -prefMapSize 266077 -jsInitLen 277128 -parentB>
├─3369479 /usr/lib/firefox/firefox -contentproc -parentBuildID 20220609170544 -prefsLen 18146 -prefMapSize 266077 -appDir /usr/lib/fire>
├─3374574 /usr/lib/firefox/firefox -contentproc -childID 161 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3473440 /usr/lib/firefox/firefox -contentproc -childID 168 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3585007 /usr/lib/firefox/plugin-container /home/matt/.mozilla/firefox/i3w1cbew.default/gmp-widevinecdm/4.10.2449.0 2707939 true gmplu>
├─3619513 /usr/lib/firefox/firefox -contentproc -childID 195 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3619611 /usr/lib/firefox/firefox -contentproc -childID 196 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3619778 /usr/lib/firefox/firefox -contentproc -childID 197 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3768700 /usr/lib/firefox/firefox -contentproc -childID 210 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3774713 /usr/lib/firefox/firefox -contentproc -childID 211 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3887801 /usr/lib/firefox/firefox -contentproc -childID 213 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3930854 /usr/lib/firefox/firefox -contentproc -childID 217 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3933460 /usr/lib/firefox/firefox -contentproc -childID 218 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3933865 /usr/lib/firefox/firefox -contentproc -childID 219 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
├─3935821 /usr/lib/firefox/firefox -contentproc -childID 220 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
└─3936239 /usr/lib/firefox/firefox -contentproc -childID 221 -isForBrowser -prefsLen 19504 -prefMapSize 266077 -jsInitLen 277128 -paren>
Jun 23 21:13:14 matt-gen-laptop-p01 systemd[809]: Started /usr/bin/firefox.
● run-re6a64a3f61e9425280a6cc758d955648.scope - /usr/bin/chromium
Loaded: loaded (/run/user/10010/systemd/transient/run-re6a64a3f61e9425280a6cc758d955648.scope; transient)
Transient: yes
Active: active (running) since Mon 2022-06-27 16:09:22 AKDT; 12min ago
Tasks: 123 (limit: 33435)
Memory: 506.6M (high: 3.4G max: 3.4G swap max: 12.0M available: 2.9G)
CPU: 3.823s
CGroup: /user.slice/user-10010.slice/user@10010.service/browsers.slice/run-re6a64a3f61e9425280a6cc758d955648.scope
├─3927358 "/usr/lib/chromium/chromium --enable-features=UseOzonePlatform --ozone-platform=wayland
├─3927371 /usr/lib/chromium/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler "--database=/home/m>
├─3927373 /usr/lib/chromium/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler "--database=/h>
├─3927377 "/usr/lib/chromium/chromium --type=zygote --no-zygote-sandbox --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash>
├─3927378 "/usr/lib/chromium/chromium --type=zygote --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Linu>
├─3927380 "/usr/lib/chromium/chromium --type=zygote --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Linu>
├─3927405 "/usr/lib/chromium/chromium --type=gpu-process --ozone-platform=wayland --enable-crashpad --crashpad-handler-pid=3927371 --en>
├─3927406 "/usr/lib/chromium/chromium --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-typ>
├─3927411 "/usr/lib/chromium/chromium --type=utility --utility-sub-type=storage.mojom.StorageService --lang=en-US --service-sandbox-typ>
├─3927482 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
├─3927483 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
└─3927558 "/usr/lib/chromium/chromium --type=renderer --enable-crashpad --crashpad-handler-pid=3927371 --enable-crash-reporter=,Arch Li>
Jun 27 16:
Sundries
thoughts on web browsers
I’d really like to see support in firefox for their tab categories. Each tab category can be restricted by a slice or unit. A clean implementation would probably require firefox speaking to systemd via dbus.
wrapping one-off commands
function wrap {
limit=$1; shift;
dispatch --name $1 \
--scope \
--slice wrap.slice \
-c 50 \
-mh $limit \
-mm $limit \
-s 1\
--binary "$@"
}
killing everything in a slice
systemctl --user stop node.slice
checking for failed commands
systemctl --user failed
check all of the user processes and their slices
systemctl --user status
some of the other slices I use
~ ls ~/.config/systemd/user/*.slice
/home/matt/.config/systemd/user/browsers.slice
/home/matt/.config/systemd/user/nvim.slice
/home/matt/.config/systemd/user/tmux.slice
/home/matt/.config/systemd/user/compilers.slice
/home/matt/.config/systemd/user/ocamllsp.slice
/home/matt/.config/systemd/user/vpn.slice
/home/matt/.config/systemd/user/gopls.slice
/home/matt/.config/systemd/user/pythonlsp.slice
/home/matt/.config/systemd/user/wrap.slice
/home/matt/.config/systemd/user/node.slice
/home/matt/.config/systemd/user/shell.slice
-
Well except for browsers and node, those get a slightly higher limit on a newer machine. ↩︎
-
I’ve spent all of my career dealing with posix environments and a good chunk in HPC. I have to say that process groups could have been fleshed out a little more fully. That fact that abstraction layers or additional automation are needed to properly manage children is a testament to that. ↩︎
-
Now why does a web browser can require that much RAM? ↩︎
-
I’m going to skip the disaster that is how I manage my virtual environments for now. ↩︎