As I'm sure a lot of you are aware, net-snmp doesn't play nicely with the reserved space on some volumes. This results in a disparity between the readings for % Utilization inside Solarwinds and those seen by admins when performing a df. Unfortunately, it appears that this is actually an issue with the way net-snmp returns data
# snmpwalk -v 2c -c public localhost hrStorage HOST-RESOURCES-MIB::hrStorageIndex.32 = INTEGER: 32 HOST-RESOURCES-MIB::hrStorageType.32 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk HOST-RESOURCES-MIB::hrStorageDescr.32 = STRING: / HOST-RESOURCES-MIB::hrStorageAllocationUnits.32 = INTEGER: 4096 Bytes HOST-RESOURCES-MIB::hrStorageSize.32 = INTEGER: 2766037 HOST-RESOURCES-MIB::hrStorageUsed.32 = INTEGER: 1000361
As you can see, net-snmp only returns Used and Size, but not available. This leaves it up to the monitoring software to perform a calculation without all of the relevant data, most notably the values available in df's Available column:
# df --block=4096 Filesystem 4K-blocks Used Available Use% Mounted on /dev/sda2 2766037 1000363 1622897 39% /
The available space is actually pulled in two different ways - bfree and bavail
statfs("/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2766037, f_bfree=1765675, f_bavail=1622898, f_files=2858240, f_ffree=2756839, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0
Taking a look at the source code for df we can see that both bavail and bfree are used.
df.c
input_units = fsu.fsu_blocksize; output_units = output_block_size; total = fsu.fsu_blocks; available = fsu.fsu_bavail; negate_available = (fsu.fsu_bavail_top_bit_set & (available != UINTMAX_MAX)); available_to_root = fsu.fsu_bfree; [..] used = total - available_to_root;
Note that bfree is assigned to 'available_to_root' while bavail is assigned to 'available'. Without getting too technical, the end result for the % used is something like this:
( (used * 100) / (used + available) ) + 1
This ends up giving a different value from what we see in Solarwinds. Based on what my research it seems that in order to obtain % used, Solarwinds is using the following formula:
hrStorageUsed / hrStorageSize
This ends up resulting in a different percentage of utilization. In this example, Solarwinds is only reporting 36% utilization opposed to the 39% shown in df. While a 3% difference may not seem like much, the problem becomes much more apparent as utilization approaches 100%:
# df --block=4096
Filesystem 4K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 1249815 890351 295976 76% /
# snmpget -v 2c -c public localhost hrStorage{Index,Type,Descr,AllocationUnits,Size,Used}.4
HOST-RESOURCES-MIB::hrStorageIndex.4 = INTEGER: 4
HOST-RESOURCES-MIB::hrStorageType.4 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageDescr.4 = STRING: /
HOST-RESOURCES-MIB::hrStorageAllocationUnits.4 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageSize.4 = INTEGER: 1249815
HOST-RESOURCES-MIB::hrStorageUsed.4 = INTEGER: 890351
However Solarwinds is showing the following information:
| Size | 4.8 GB | |||
| Space Used | 3.4 GB | |||
| Space Available | 1.4 GB | |||
| Percent Used | 71 % |  | ||
| Percent Available | 29 % | |
At this point we are looking at a 5% difference. As you approach 100% the difference becomes even greater. We first discovered this problem ourselves when the 95% threshold alerts that we had set up never alerted even though the volumes had completed filled to the point that applications ceased functioning.
Unfortunately, we are still left with the problem of how to properly monitor these volumes. I don't believe that Solarwinds can modify the formula because Net-SNMP only provides total and used, so we are left with modifying other aspects.
One method would be to arbitrarily reduce the threshold; however, we can't be sure that all volumes are limited to 5% reserved space. A second method (which we have pursued) is to implement script monitors for all volumes - We started running into issues with this as the component count for these monitors quickly approached 1500 with more being added every week. Our last course of action is to modify the way that Net-SNMP sends data for these devices. To that end I've started to come up with a possible solution; however, having never dealt with something like this before I'd appreciate any advice.
snmpd.conf gives you the ability to implement extension commands that can be used to change the data before it is sent out. My thinking here is that this can be used to pre-shrink the total size by the reserved space. This would have the result of bringing the results in Solarwinds closer to what is shown in df, except it appears that we will still be 1% off. This should be enough though to allow us to properly establish monitoring and alerting on Linux volumes through SNMP. The pass would be setup on HOST-RESOURCES-MIB::hrStorageSize and use the information available in `df --block=4096` to determine the new size:
# df --block=4096 Filesystem 4K-blocks Used Available Use% Mounted on /dev/sda2 2766037 1000363 1622897 39% / /dev/sda5 254802 49519 205283 20% /var
The new size would be found by using the following formula:
Used + Available = Size
This ends up giving us a fairly accurate result whether we are at the default 5% reserved space (/) or at 0% reserved space (/var)
/
1000363 + 1622972 = 2623335 2623335 / 2766037 = 0.948
/var
49519 + 205283 = 254802
Now when Solarwinds runs the used / total formula we get the following result:
1000363 / 2623335 = 0.381 49519 / 254802 = 0.194
These values are only 1% off from what is shown by df. The same holds true for the higher utilization shown before:
890351 + 295976 = 1186327 890351 / 1186327 = 0.750
The reason for my lengthy post is that I've never done anything like this before and was hoping for some advice on how to proceed by anyone that has used an SNMP extension before. If nothing else, I'm hoping that my research and progress will benefit others.
I'm thinking of testing with a pass then actually implementing using persist pass. The actual script will be run through Perl and it will have to be deployed to ~400 different servers. What kind of testing would need to be involved since most of these are production devices, although backing out should be as simple as restoring the backed up snmpd.conf file and removing the .pl. Also, is pass/persist pass the best way to go about this? From my reading I don't think that exec or extend will allow me to completely override hrStorageSize - These seem to be more for additions than replacements. Any caveats I need to worry about when implementing the script? I've already noted the empty line when shutting down and PING/PONG handshake for pass persists.
One major hurdle I still need to cross is how to get the description of the volume with just the index and not having to resort to another snmpget just for this. Any advice on getting this is much appreciated.
Thanks in advance,
Bob
