Fixing LabTech Patching – Find Online Machines With ‘Unknown’ Patch Inventory

This is the first in a series of posts about fixing LabTech patching. This came up with the Wannacry ransomware that caused all partners to take a closer look at their patching setup.

Many of our machines are showing no patch inventory present, but are installing patches anyway. Even more interesting, these computers have run patch jobs. The scores aren’t updating and the last patch Inventory is Unknown:

Working with support, we found that clicking Resend to force an update to the inventory will solve the problem for that machine. However, I wanted to find a way to automate this.

NOTE: A previous version of this post incorrectly referenced the computerpatchcompliancestats table. While that information is available there, it requires finding all rows that are in one table and not another. Additionally, computerpatchcompliancestatsgets regenerated by the database agent once per day and won’t be up to date after a successful scan.

Information about when the last inventory was scanned is stored in the WindowsUpdate column of the computers table.

To find machines with this problem, look for rows without any value for the WindowsUpdate column. Note that this isn’t NULL but an empty string:

You might also find machines that instead of having an empty have the epoch listed. Checking the database tables for this issue, some will be: 1600-12-31 19:00:00 (I had to look this up as I hadn’t seen that as the epoch before). Ones that have this value are likely broken and will require more than this script to fix (see the next post).

Next, create an alert template against this monitor that runs a script to execute the resend patch information command.

This is a one-line script that updates the inventory:

This has made short work of the ‘No Inventory’ problem. You can use a similar issue to address machines that haven’t had a patch scan in a certain interval of time because of the configured schedules.

ZFS Error on Datto Upgrade

As we continue upgrading our Datto fleet to 16.04, we’ve encountered some problems.  A new one this morning:

After reboot, the device was checking in, but the zfs filesystems weren’t mounted.  This happens occasionally, even during normal operation.  However, I wasn’t able to execute any zfs commands to fix it.  I received this, even with a basic one:


root@datto:~# zfs list
The /dev/zfs device is missing and must be created.
Try running 'udevadm trigger' as root to create it.


This is a misleading error message. As noted on this GitHub Issue, the issue is that /etc/mtab is missing. Recreate with


root@datto:~# touch /etc/mtab


Then reboot.

Datto Upgrade Error

In upgrading devices to Ubuntu 16.04, I’ve encountered a problem with / more than once now. Specifically, it’s coming up read only:


root@datto:~# checkin
Checking to see if checkin is currently running
/datto/scripts/DoBackup.sh: line 92: /datto/config/checkin.lock: Read-only file system
flock: 200: Bad file descriptor
Checkin is currently running. Exiting...


The checkin script uses a lock file as a semaphore to ensure that only one copy of the script executes at once. Even though there aren’t any other instances running, the checkin script fails because it can’t create the semaphore, with the self-explanatory error of “read-only file system”.

The output from mount verifies this:

/dev/md1 on / type ext4 (ro,relatime,data=ordered)

As the error indicates, root is mounted read-only. Obviously not the ideal situation.

The first, obvious, step is to simply make / read/write:


root@datto:~# mount -o remount,rw /
mount: can't find UUID=eaf3a162-d770-4ec9-a819-ec96d429ea9f


The device name of /dev/md1 indicates that this is a software RAID volume. What seems to happen during these upgrades is that the UUID changes, meaning that the entry for / in fstab is wrong. To fix this, first get / mounted read-write:


root@datto:~# mount -o remount,rw /dev/md1 /
root@datto:~#


Checkin will work now. If you can get the device to checkin again, this is a good time to call support to verify that there are no other issues from the upgrade.

To be clear, that’s my recommendation: Call support once you get the deice to checking in again. Most partners shouldn’t be messing around with the appliances like the below.

Note that the web won’t work yet. To bring that up, issue

root@datto:~# /etc/init.d/php7.0-fpm restart
root@datto:~# /etc/init.d/apache2 restart


You’ll also find other problems on the device (the zfs pools are unmounted). While you can fix these manually, your efforts will go to waste once the device reboots. To fix this permanently, you need to fix the UUID for / in fstab.

Make a backup copy of fstab first:

root@datto:~# cp /etc/fstab /etc/fstab.bak


Then, identify the correct UUID for /dev/md1

root@datto:~# blkid /dev/md1
/dev/md1: UUID="13152fae-d25a-4d78-b318-74397eb08184" TYPE="ext4"


Then, edit fstab and change the line for md1 to use the new, correct UUID.

Next, ensure that there is a valid mtab file to use:

root@datto:~# touch /etc/mtab