Given it’s the holidays and that’s when I tend to have time and energy to poke around the house, I decided to upgrade my FreeNAS box to the newer TrueNAS 12 release. The upgrade itself went without a hitch, simply switching from the FreeNAS 11 train to the TrueNAS 12 train was sufficient. However when everything came back, none of my jails started. I have three jails that are used daily: plex, gitea, and inkdrop (which runs a CouchDB server for the Inkdrop note-taking tool I’ve detailed before).
Noting this here, as a followup to my earlier blog post about updating iocage jails in FreeNAS was helpful but insufficient. It might have been sufficient on any day other than today though…
Trying to start the jails in the TrueNAS UI yielded no real information as to why the jails weren’t starting, so off to the trusty commandline:
# sudo iocage start inkdrop
No default gateway found for ipv6.
* Starting inkdrop
inkdrop devfs_ruleset 7 does not exist! - Not starting jail
Still not overly helpful, but some poking around led me to this issue on GitHub that described the problem I was having. You could edit the config.json file directly, but iocage let’s you use tooling for this as well.
# sudo iocage get all inkdrop|grep dev
allow_mount_devfs:0
devfs_ruleset:7
min_dyn_devfs_ruleset:1000
mount_devfs:1
# sudo iocage set devfs_ruleset=4 inkdrop
devfs_ruleset: 7 -> 4
Here we set the devfs_ruleset to 4. Why 4? No idea, that’s what the GitHub issue noted to set it to. Also note that these were incremental; my plex server had a devfs_ruleset value of 5, gitea was 6, and inkdrop was 7 (incidentally the order in which they were created). I set all of them to 4. Now the jail starts:
# sudo iocage start inkdrop
No default gateway found for ipv6.
* Starting inkdrop
+ Started OK
+ Using devfs_ruleset: 1001 (iocage generated default)
+ Configuring VNET OK
+ Using IP options: vnet
+ Starting services OK
+ Executing poststart OK
+ DHCP Address: 192.168.1.82/23
Now that the jail started, I pulled in the FreeBSD 12.2-RELEASE with iocage fetch. And following previous instructions I had outlined wanted to do the upgrade. Which would have worked, except that the FreeBSD folks appear to be monkeying around with their git repos today! Since I wanted to get this all done today rather than wait, I decided to do some more digging.
The upgrade error:
# sudo iocage upgrade inkdrop -r 12.2-RELEASE
Traceback (most recent call last):
File "/usr/local/bin/iocage", line 10, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/iocage_cli/upgrade.py", line 40, in cli
ioc.IOCage(jail=jail, skip_jails=skip_jails).upgrade(release)
File "/usr/local/lib/python3.8/site-packages/iocage_lib/iocage.py", line 2070, in upgrade
new_release = ioc_upgrade.IOCUpgrade(
File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py", line 112, in upgrade_jail
with urllib.request.urlopen(f) as fbsd_update:
File "/usr/local/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/local/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/local/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/local/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
The patch I applied, based on this issue in GitHub detailing the same problem:
# diff -uN ioc_upgrade.py /usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py
--- ioc_upgrade.py 2020-12-22 11:45:17.715773034 -0700
+++ /usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py 2020-12-22 11:45:58.144014593 -0700
@@ -104,6 +104,8 @@
f = 'https://raw.githubusercontent.com/freebsd/freebsd' \
f'/release/{f_rel}/usr.sbin/freebsd-update/freebsd-update.sh'
+ f = 'https://raw.githubusercontent.com/freebsd/freebsd/master/usr.sbin/freebsd-update/freebsd-update.sh'
+
tmp = None
try:
tmp = tempfile.NamedTemporaryFile(delete=False)
Now performing the upgrade worked:
# sudo iocage upgrade inkdrop -r 12.2-RELEASE
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 11.3-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata files... done.
Inspecting system... done.
The following components of FreeBSD seem to be installed:
world/base world/doc world/lib32
The following components of FreeBSD do not seem to be installed:
world/base-dbg world/lib32-dbg
Does this look reasonable (y/n)? y
...
Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run "/tmp/tmp723gk2ae install"
again to finish installing updates.
src component not installed, skipped
Installing updates...rmdir: /mnt/storage/iocage/jails/inkdrop/root//var/db/etcupdate/current/usr/share/openssl/man/en.ISO8859-1: Directory not empty
rmdir: /mnt/storage/iocage/jails/inkdrop/root//var/db/etcupdate/current/usr/share/openssl/man: Directory not empty
rmdir: /mnt/storage/iocage/jails/inkdrop/root//var/db/etcupdate/current/usr/share/openssl: Directory not empty
rmdir: /mnt/storage/iocage/jails/inkdrop/root//var/db/etcupdate/current/usr/share/man/en.UTF-8: Directory not empty
rmdir: /mnt/storage/iocage/jails/inkdrop/root//var/db/etcupdate/current/usr/share/man/en.ISO8859-1: Directory not empty
rmdir: /mnt/storage/iocage/jails/inkdrop/root//var/db/etcupdate/current/usr/share/man: Directory not empty
done.
inkdrop successfully upgraded from 11.3-RELEASE-p14 to 12.2-RELEASE-p2!
Now that the upgrade was completed (which took a bit), time to log into the jail and do a userland upgrade:
# sudo iocage console inkdrop
Password:
Last login: Tue Dec 22 11:24:12 on pts/0
FreeBSD 12.2-RELEASE-p2 663e6b09467(HEAD) TRUENAS
Welcome to FreeBSD!
...
root@inkdrop:~ # pkg update && pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking for upgrades (77 candidates): 100%
Processing candidates (77 candidates): 100%
The following 77 package(s) will be affected (of 0 checked):
....
Seems to be working well! The web UI now shows the inkdrop jail running 12.2-RELEASE-p2 which is what I wanted. Rinse and repeat for the other two jails. Both of which worked just as well. All in all, upgrading the jails took longer than upgrading from FreeNAS to TrueNAS.
So far so good! I still have to upgrade the ZFS pools, and FreeNAS (sorry, TrueNAS… will take some getting used to) also supports being an OpenVPN server which is something I’ve missed since I switched from pfSense to the Unifi USG. I’ll have to fiddle with that later on.
If it wasn’t for the, hopefully transient, changes to the FreeBSD git repos it likely would have all been fine. Judging by the GitHub issues, if I had done this last weekend it probably wouldn’t have been an issue. If you do opt to do like me, make sure you have a backup and probably take snapshots of your jails first so you can rollback if you have any problems (which, as I’m typing this, made me think that would have been a good idea that I neglected to do myself…). I’ll be fiddling with this new TrueNAS release over the next few days, but so far it seems pretty good.
Also don’t forget to revert the changes to ioc_upgrade.py when you’re done!