Logging NFS Server Activity in LINUX

I recently ran across this problem. I had a situation where files being accessed were being deleted by another system over the NFS protocol. With FTP and SMB/CIFS access, I could just enable greater logging and track the problem down, but with NFS server software on LINUX, there isn't this ability.

I researching it, I was able to determine that "tshark", a part of the Wireshark project, had the ability to capture and print out decoded NFS protocol packets. Running "tshark -R nfs" on the local server allowed me to see all NFS traffic on that local server. The output is formatted to be very readable and I could easily see the REMOVE packets that the remote server was sending, and from that I was able to determine which system was deleting files.

Since my problem seem to be related to an automated process starting at 11:30pm, I didn't have to run tshark for very long but the log was still over 4 Megabytes. If a longer monitoring time was required, I could have just run the following command: "tshark -R nfs | grep REMOVE >/tmp/REMOVE.log"
This would have only stored away NFS file deletions.

Hopefully, this posting will help someone in the future doing a search for a LINUX NFS Log ability (which right now leads to postings that say it can't be done).

Windows Active Directory Integration pitfalls

OSVault integrates into a Microsoft Windows Active Directory environment, but sometimes the SAMBA integration with user permissions can be challenging. For example, unless it is explicitly setup, the Administrator account does not always have unlimited permissions to change the security settings on a file stored in OSVault. This is because Administrator privileges are not always translated to "root" privileges in the OSVault/SAMBA software.

The problem comes from custom environments. Usually, the Domain Administrator is called "Administrator" and is in the group "Domain Admins". So, in the OSVault system, we map those user/group permissions to "root" permission. But if you change the names of that user or that group, OSVault won't know it, and will treat those users as a regular (non-privileged) user.

The file /etc/samba/smb.conf in the OSVault/InfiniDisc appliance has the following lines in it:

winbind separator = +
inherit acls
map acl inherit
username map = /etc/samba/smbusers

And in the Share portion
admin users = @"DOMAIN+Domain Admins", DOMAIN+Administrator

Notice that the "+" in the admin users line is the same character as in the winbind separator line. That is VERY important and not mentioned in a lot of on-line material

XFS B-Tree error and Building a New DMAPI kernel

In LINUX Kernel releases prior to, and including 2.6.29, a large number of file creations and deletions in a single directory uncovers a bug that causes the XFS file system to become unavailable. This is evidenced by the following error message in the system log (/var/log/messages):
infinidiscdev kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 3327 of file fs/xfs/xfs_btree.c. Caller 0xc032b241

This only occurs on 32-bit InfiniDisc systems, and not on any installed OSVault systems. So far the problem has only been seen in lab testing using the BackupPC software to backup a large number of client systems to an InfiniDisc system.

Since nearly all of our customers for OSVault archive systems do not usually do large amounts of file deletions, this problem would be hard to find in that environment.

The fix is to update to a newer kernel version, such as 2.6.31 or 2.6.33. Our InfiniDisc systems will update to 2.6.33, which we get from XFS.ORG but then have to make a couple of patches to. The reason for the patches is that we have legacy systems built with gcc 3.4 and the baseline 2.6.33 kernel will not compile with that kernel. For our more current InfiniDisc appliances built on CentOS 5.4, the changes to the standard kernel source are not required.

The primary change we make to 2.6.33 is in the mptsas message routine, where we move the mptsas_set_rphy routine before the mptsas_port_delete routine.