CIFS to DFS and Netapp Filer performance problems

[Yosemiteuser]

Hello,
probably this, or a very similar scenario, has been posted before therefore sorry for asking again.

Let my describe my scenario (Windows clients are working fine, therfore will be out of the scope)

Netapp Filer with ONTAP 8.2.2 7-Mode acting as the filer server, Volume and Share with NTFS security style.

Microsoft DFS running in a Windows 2008 R2 Server with ABE (Access based enumeration) enabled

MAC clients with Yosemite 10.10.5 not joined to our Windows Domain, connecting to the DFS namespace using cifs://namespace/folder and after that entering the Windows domain credentials.

The problems we are facing :

Some Macs computers are very slow when browsing the share folders (probably ABE is not helping in this point)

Some Macs are just working "fine" (slow but stable) but after some hours of inactivity (1 hour pause for example) the share becomes disconnected from the computer.

Some Macs suffers both problems, slow and erratic disconnections of the share

What we have checked/tried :

Ethernet seetings seems to be fine. Primary and secondary DNS Servers are our main Domain Controllers, Domain name added to the Domain Search list, Workgroup name like our Domain name. Network link is in Auto but it must be 1Gb link.

Time sinchronization is done with our main Domain Controller

smbutil statshares -a reflects that the share connection is, by default, done using SMB 2.1 (which we think it works faster but more erratic and unestable than SMB 1)

smbutil dfs shows that it seems there is no error in our DFS referals

Forcing SMB to be SMB1 in the snmb.conf file seems that makes the connection more stable (and slower of course) but disconnection of the share still happen after some period of inactivity.

Energy saving settings habe been reviewed forcing the computer to be aways alive. With a continuous ping to a Mac computer we saw that the Ethernet connection was always working while the share was disconnected, therefore probably is not a network problem.

With klist we see that connecting with the DFS will not make a Kerberos ticket creation (don´t sure it this will help to speed up or mantain the share connected). Connecting directly to the Netapp share will create a Kerberos ticket. Why this is working different between the two connection methods? (unfortunately we need to use DFS and ABE).

What can we do?

Is there any other thing we can try? Any other tweak in the snmb.conf file? My first option was to think in the Neatpp Filer but as far as it seems that the share is disconnected from the DFS namespace, could it be posible that is there something that must be checked in the DFS side?

Thanks in advance for your help and answers.

[jhalvorson]

For our computers with 10.10 or earlier, I've offered a script via a Self Service policy that adjusts the TCP Delay Ack to 2. (Apple default is set to 3). This has helped with the speed of browsing the file shares.

#!/bin/sh

################################################################################
#
#  This script set the TCP delay ack to 2, as recommended by Isilon to help improve the
#  performance of browsing, saving, and reading from network file shares.
#
#  REFERENCES
#  PDF document:  Using Mac OS X Clients with Isilon OneFS 6.5 (Oct 2012)
#  http://hints.macworld.com/article.php?story=20051107090652912
# 
#       Modified for use at for use at company_name by Jason Halvorson
#       Aug 8, 2013
#  
#################################################################################

##########################################################################################
# Create time date stamped log to record actions taken by this script.
##########################################################################################
logFile="/private/var/log/companynameIT.log"

log () {
    /bin/echo $1
    /bin/echo $(date "+%Y-%m-%d %H:%M:%S: ") $1 >> $logFile 
}

log ""
log "BEGIN routing of changing the TCP Delayed Acknowledgment to 2 for better performance"

##########################################################################################
#  STEP 1
#    The following line sets the delay for the current user logged in and works 
#    immediately, but reverts back to 3 (auto) after a reboot.
sudo sysctl -w net.inet.tcp.delayed_ack=2
log "Changed the users sysctl to tcp delay ack 2"

##########################################################################################
#  STEP 2
#    Clear any previous sysctl.conf file
if [ -f "/private/etc/sysctl.conf" ]; then
    log "Deleted the existing sysctl.conf file " 
    /bin/rm -f "/private/etc/sysctl.conf"
fi

#    Set the delay to 2 so that it remains after a reboot
sudo bash -c "echo 'net.inet.tcp.delayed_ack=2' >> /etc/sysctl.conf"
log "Changed the etc/syctl.conf to tcp delay ack 2"


log "COMPLETED changed the TCP Delayed Acknowledgment to 2."

exit 0

I also have the same script and another Self Service that sets the value to 3 in case people found that setting it to two did not help.
I haven't been asked to allow this script to run on 10.11.x or 10.12.x, so that leads me to believe the files sharing speeds are OK with the newer OSes.

[Yosemiteuser]

Hi Jason,

many thanks for your answer. I made the change in some pilot users and I´m waiting for a confirmation about the browsing folders improvement.

I saw that other people changed the value to 0 instead to 2. Did you tried also this value?

Regarding DFS, do you have disconnections of the share in the Mac side after some time of inactivity? Is there any tweak or best practice in the DFS side that you can share with me?

In my scenario, my DFS Server is a clasical instalation of Windows Server 2008 R2 , and as far as I think that ABE will make things slower for the clients, do you think that improving the RAM memory or the CPU in the DFS Server I will get more stabilty in the MAC share connection?

Thanks in advance for your help.

Regards

[jhalvorson]

If I recall, I went with "2" because that was mentioned in the Isilon OneFS 6.5 documentation.

The disconnections are usually related to when the Mac goes to sleep and the network connections drop. I believe the reconnect is much better or more likely to occur with 10.11 or later. I don't recall getting much feedback from users.

It's another team that manages the servers. I recall a document or someone from the other team determined best way to improve the performance was by adding SSD to host the caches on the filer. It's been a long time since I've worked this issue. Not really hearing much complaints now a days. I don't know the details of our current file server environment.

[Yosemiteuser]

Hi,
sorry for my late update, it has been a hard week.

Sleep options are disabled, this is the first we checked as far as we also believed that this was the cause.

So, this is the update :

ACK=2 seems to improve a little bit the speed, ACK=0 improves even a little bit more but as far as we are "forced" to use SMB v.1 there is not much we can do with speed.

After many tests and comparing same client connected to a Netapp filer share using and not using DFS , with the "klist -v" command we saw that when using DFS there was no kerberos involved in the connection and therefore the shares where disconnected randomly.

What we saw is that when connecting to the dfs namespace cifs://mydomain.net/namespace/dfsfolder/dfsfolder there was no kerberos negotiation in the first step and therefore the rest of the connections to the Netapp Filer when browsing the folder targets were also done without a Kerberos ticket.
If we changed the connection to force the Mac to use a specific DFS Server using cifs://mydfsserver.mydomain.net/namespace/dfsfolder/dfsfolder we got a kerberos ticket and somehow the connection to the share became more stable until the point that in the pilot Mac client it was connected for all the day whitout disruptions.

From that point, what we did is to force the creation of a kerberos ticket during the log on of the user using a script with something inside like "kinit -l 10h --renewable USER@MYDOMAIN.NET" (user has to enter the password in the terminal window). After the ticket creation we manually connect to the share and it must be using the kerberos ticket because the user is not prompted again for credentials to connect to the share.

I don´t know if this is the best solution but it has been a good improvement for us with the share mounting stability. Of course using this solution we loose the "high availability" options of DFS because we have more than one DFS Server and forcing to use always the master server in the cifs connection, if this server goes down Mac clients will not be able to reach the folders while the secondary DFS server will not be used. To solve this I think I can make a DNS alias like MYDFSFORMACS and use it in the cifs connection to make it point to the DFS server I want to use and if for some reason my master DFS server is not working I can easily and quickly change the alias pointing to the secondary DFS server and recover the situation without having to reconfigure the cifs connection in every Mac computer.

Will be good to know also if there is some way to create kerberos tickets longer than 10 hours whichs seemt to be the default.

Next steps will be to try to upgrade ONTAP from 8.2.2 to 8.2.3 and see if CIFS is able to negotiate something higher than v.1 because after the change in the connection method we saw that only v.1 was negotiated (not 100% sure if this is due to the W2K8 DFS server).

I will keep this thread updated and of course, any help or comments will be really apreciated. Hope this also helps somebody.

Regards.

[jhalvorson]

Take a look at Kerbminder as a possible solution for assisting with the kerberos tickets.