I have received a lot of questions from customers lately on whether there is a need to tune Windows Server 2008 file servers, especially if the client machines connecting to them are capable of using SMB 2.0, which includes Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2. Unfortunately, there is not much information out there on this topic, so I decided to do a little research and testing for myself.
To begin, it is important to understand the SMB 1.0 limitation that has been present since we first started implementing Terminal Server deployments back in the good old days of NT and MetaFrame. There have already been a lot of good SMB tuning articles out there that discuss this in detail, so I will not go into too much of an extensive discussion here, but I will summarize.
When a client computer using SMB 1.0 (NT 4.0, 2000, XP, 2003, etc…) attempts to connect to a Windows file server, it will query the file server and ask how many concurrent network (SMB) commands it can have submitted and open simultaneously. The file server will respond with a number and the network redirector on the client computer will limit itself to the number provided by the file server. The number that the file server provides is controlled by the SMB value for Max Mpx Count, which is set by the following registry entry on Windows file servers:
By default, this value does not exist on a Windows Server. If it does not exist, the Operating System uses a default value of 50.
This means that an individual client computer will not be able to have more than 50 simultaneous SMB commands to the file server. An SMB command can be anything from a directory listing, file creation, deletion, ACL manipulation, etc… basically, any kind of file or directory access.
The 50 command limitation quickly becomes a problem on a Terminal Server because there is only one redirector that is shared by all users on the server. In a typical Terminal Server environment, often users will all connect to the same file server for home directories, roaming profiles and redirected folders. This means that each user could easily be generating multiple SMB commands to a single file server. Once you start loading 50+ users on the server, you can easily have more than 50 outstanding SMB commands that need to be serviced, especially if folder redirection is being used. Since only 50 get serviced at one time, the rest of the commands begin to queue up and wait for servicing. This can cause poor performance or even application failures as applications make file requests that time out waiting to be serviced.
A workstation or SMB Client may make a request to have more than 50 SMB commands simultaneously open to a file server; however, if the file server has not been tuned, the client will not use more than the Max Mpx Count returned by the file server. The maximum number of simultaneous requests that a workstation will attempt to use is controlled by the Maximum Commands SMB setting defined by the following registry entry on the client:
By default this value is also 50 if it is not defined. So to properly tune an environment for maximum density of Terminal Server/XenApp users, you must add the MaxCmds key to all Terminal Serves and the MaxMpxCt key to all file servers. We have typically recommended increasing these entries to a decimal value of 2048.
Microsoft has a good article (324446) that has been around since Windows 2000 that discusses this issue and recommends the follow registry keys that should be implemented on file servers being accessed by Terminal Servers:
“MaxWorkItems”=dword:00002000 (decimal 8192)
“MaxMpxCt”=dword:00000800 (decimal 2048)
“MaxRawWorkItems”=dword:00000200 (decimal 512)
“MaxFreeConnections”=dword:00000064 (decimal 100)
“MinFreeConnections”=dword:00000020 (decimal 32)
It is also important to note that this issue and these registry keys are fully applicable to Terminal Servers and File Servers running the x64 edition of Window Server 2003. This is not a 32-bit problem, but rather an SMB problem.
In addition to the above registry tuning, there was another SMB client fix which reduces the SMB chatter and SMB commands that are opened from a client to a file server. This registry setting is one that we recommend be implemented on all XenApp Servers:
You can verify whether or not your system has excessive SMB commands being queued by reviewing the following performance monitor counter on your XenApp servers: Redirector/Current Commands. If this counter gets close to 50 and you have not tuned your file servers, then you definitely have a problem.
Here are some Microsoft articles for reference:
Before we dive into a pure SMB 2.0 environment, it is important to understand what happens in a mixed environment. Consider the following scenarios:
In this scenario, the XenApp servers are only capable of using SMB 1.0, so even though the Windows 2008 Servers support SMB 2.0, the session from XenApp on Windows 2003 will be SMB 1.0. This means that both the XenApp servers and Windows 2008 file servers need to be tuned. I used Wireshark to capture and view an SMB session from a Windows 2003 Server running XenApp opening a file from a Windows 2008 R2 file server.
The following screen shot is the SMB Negotiation request that was sent from the Windows 2003 Server to the Windows 2008 R2 server.
You can see under the Requested Dialects section all the versions of SMB that Windows 2003 Server supports. The last dialect listed, NT LM 0.12, is SMB 1.0.
Following is a screen shot of the SMB response sent from the Windows 2008 R2 file server to the Windows 2003 XenApp server:
You can see that the Windows 2008 R2 file server agreed to use “NT LM 0.12″ as the SMB dialect, which is SMB 1.0. Also, take note of how the highlighted line shows that the Max Mpx Count for this SMB session will only allow 50 simultaneous commands. The Windows 2003 XenApp server has the MaxCmds registry setting set to 2048; however, the Windows 2008 R2 file server is a default install and the MaxMpxCt registry key on the server does not exist. This shows that Windows 2008 R2 still uses the SMB 1.0 default value of 50 when Windows 2003 servers connect to it via SMB.
For the next screen shot I tuned the Windows 2008 R2 file server to use all of the recommend values that we typically implement on Windows 2003 file servers based upon MS article 324446 and rebooted. I then connected from the Windows 2003 XenApp Server to the tuned Windows 2008 R2 file server.
Now that the Windows 2008 R2 server has been tuned, the Max Mpx Count is now increased to 2048. This clearly shows all of the SMB tuning parameters from Microsoft article 324446 should be implemented on Windows 2008 R2 file servers.
This brings us to the other mixed scenario where Windows Server 2008 Remote Desktop Servers with XenApp connect to Window Server 2003 file servers. In order to see what happens in this scenario, I took a standard install of Windows Server 2008 R2 and connected to a Windows Server 2003 file server that has been fully tuned per MS article 324446.
The screen shot below shows the SMB negotiation request sent from Windows 2008 R2 to a Windows 2003 file server.
You will notice there are two additional SMB dialects requested as compared to the Windows 2003 XenApp Server. The “SMB 2.002″ dialect is the version of SMB that shipped with Windows Vista and Windows Server 2008. Microsoft made some improvements to SMB 2 with Windows 7 and Windows Server 2008 R2. These improvements are part of the request for SMB “2.???“. The reason for using question marks is so that additional versions of SMB 2.x do not need to explicitly be listed as a new dialect. Microsoft enhanced the dialect for “SMB 2.???” in such as way that the client and server will auto negotiate the proper 2.x revision of SMB 2 without needing to explicitly call out a new dialect in the SMB negotiation request.
The following screen shot is of the SMB response from the Windows 2003 file server to the Windows 2008 R2 XenApp server.
Take note of how the SMB 1.0 dialect was selected and how the tuned Window 2003 file server told the 2008 R2 XenApp server that it can have 2048 outstanding SMB requests based on the Max Mpx Count.
The following screen shot is of the SMB session setup response sent from the Windows 2008 R2 XenApp Server to the Windows 2003 file server.
Take note of how the Windows Server 2008 R2 XenApp server reduced the Max Mpx Count value to 50 despite the fact that the Windows 2003 file server advertised a value of 2048. This shows that an out of the box Windows 2008 R2 XenApp server is still limited to a maximum of 50 outstanding SMB commands when connecting to a file server using SMB 1.0. In order to address this, the MaxCmds value on the Windows Server 2008 R2 XenApp server still needs to be tuned.
So this brings us to the question of whether we need to tune a pure SMB 2.0 environment. Specifically, Windows Server 2008 XenApp servers connecting to Windows 2008 or later file servers using SMB 2.0. In order to answer this question we need to know what has changed between SMB 1.0 and SMB 2.0. There are many improvements in SMB 2.0, but the key improvements that will affect XenApp deployments are listed below:
- SMB 2.0 command consolidation. SMB 2.0 commands have been enhanced to provide greater functionality with fewer commands. Additionally, multiple commands can be compounded together into a single request. What might have taken three separate commands under SMB 1.0 can now be collapsed into a single request. This greatly reduces the amount of open commands that the client needs to submit.
- SMB 2.0 Sessions are per user NOT per computer. This is probably the single greatest improvement from a XenApp perspective. Under SMB 1.0, all users on a single XenApp server shared a single SMB session to the file server and had to share the Max Mpx Count limit set by the server. With SMB 2.0 each user gets their own independent SMB session with only their own open command requests counting against their maximum amount allowed.
- Max Mpx Count has been replaced with credits. The Max Mpx Count concept has been replaced with credits. The server and the client can dynamically scale the amount of credits that an SMB session can consume based upon load and network conditions. One credit roughly equates to one open command request, which could be a compounded request with multiple commands. The credit concept also provides the ability for certain types of command operations to cost more credits if they are more resource intensive. Since each user on a XenApp server has their own SMB session, one user’s submission of commands or use of credits, will not affect another user’s SMB session credits. By default, each user may consume a maximum of 128 credits, which should be more than enough for most XenApp users.
Given the above improvements which are natively part of SMB 2.0, there should not be any need to tune the SMB 2.0 settings specifically in support of XenApp deployments. With that being said there are several important registry keys to be aware of that can and probably should be tuned for any Windows 2008 R2 file server. In fact, for large high usage Windows 2008 file servers with multiple cores, high-throughput NICs and fast storage, there are some tuning parameters that you should implement. Some of the more important keys are detailed in the next section.
- TreatHostAsStableStorage (REG_DWORD). This key does not exist by default and has an assumed of value 0. If your file server provides storage that is protected with battery backed write cache, then this can be set to 1. Setting this to 1 will cause the server to ignore write flush operations initiated from client computers. This will improve performance and reduce overhead on the file server.
- MaxThreadsPerQueue (REG_DWORD). This key does not exist by default and has an assumed value of 20. On servers that support a large number of users and have multiple processors, this value should be increased to 64 decimal.
- Smb2 (REG_DWORD). This key does not exist by default and has an assumed value of 1. Settings this to zero will disable SMB 2 and force the server to use SMB 1.0. It is recommended that this key be set to 1 to ensure that SMB 2.0 is enabled.
- Smb2CreditsMin (REG_DWORD). This key does not exist by default and has an assumed value of 64. This value determines the minimum amount of credits to which a file server will forcibly limit an SMB session. This key should not need to be modified.
- Smb2CreditsMax (REG_DWORD). This key does not exist by default and has an assumed value of 1024. This value determines the maximum amount of credits a file server will allow for a single SMB session. This key should not need to be modified.
- AdditionalCriticalWorkerThreads (REG_DWORD). The default value is 0. On high usage file servers with multiple processors, especially 4 or more, you can add additional kernel threads that can be used by the System Cache to improve storage performance. This value should be increased to 64 decimal.
Below are some important new registry keys that can be tuned on an SMB 2.0 Client computer.
- DisableBandwidthThrottling (REG_DWORD). This key does not exist by default and has an assumed value of 0. The SMB 2 client will try to limit its network throughout on links that it perceives as latent. Since most XenApp deployments should be mapping drives to file servers that are highly connected, there is no need to try and limit the throughput of the SMB sessions. This value should be set to 1.
- MaxCredits (REG_DWORD). This key does not exist by default and has an assumed value of 128. By default a single user’s SMB 2.0 session will be limited to 128 credits. This is similar to the old MaxCmds settings under SMB 1.0. However, since each user gets their own queue of 128 credits, there is not typically a requirement to increase this value unless there are network intensive applications running off file servers. There may be a need to increase this value when running the App-V shared cache from a UNC path. More research will need to be done to determine an optimal value. However, since the file server by default supports up to 1024 credits per session, this value may need to be increased.
I would like to thank Ralf Schnell and Jim Pinkerton from Microsoft who provided me with some great PPT presentations that they put together on SMB 2. There is also a good article from Microsoft that discusses performance tuning of Windows Server 2008 R2. You can find the article at the following link:
I hope you find this post useful!