Recently, I get chance to upgrade and migrate my existing environment from vSphere 5 and 5.5 to version 6. During migration I have seen very strange issue in VMware TCP/IP Stack configuration. I am working on it from two days and finally I am able to figure out where the problem exists.
Here is my scenario. I have been trying to configure vMotion, NFS and Mgmt traffic on the single 10 G Ethernet. In which Mgmt is on layer 3 network. However NFS and vMotion is on layer 2 network.
I have read about the TCP/IP stack feature in vSphere and figure, let’s give him a try and use it to separate my vMotion and NFS traffic on different stack from the default TCP/IP stack. The default stack is by default used for Management traffic during the installation and configure of ESXi.
Those who used vSphere 6 will know there is a TCP/IP stacks exists for vMotion and Provision traffic on web client. You can see it from web client by going into the ESXi networking.
You have ability to create your own custom stack. but its only used for IP based storage such as ISCSI and NFS.
NOTE: Unfortunately, custom TCP/IP stacks aren’t supported for use with fault tolerance logging, management traffic, Virtual SAN traffic, vSphere Replication traffic, or vSphere Replication NFC traffic.
If you see I have created one custom TCP/IP stack for NFS traffic.
NOTE: At the time of this post. There is only one way to create custom stack by using esxcli command. Here is syntax of the command.
#esxcli network ip netstack add –N “NFS”
If you want to see vmKernel adapters association with existing stacks. Here you can see in below screen it did not have any association right now. Let’s use them.
There is only one way to use this stack. Which is at time of creation of vmKernel adapter. Let’s create one vmKernel adapter for NFS and vMotion.
To add the vmKernel adapter in ESXi server.
From web client -> select the ESXi Hosts->Manage -> Network->VMKernel Adapters-> Click on “Glob sign” to add a new vMKernel adapter.
When wizard started. During vMotion vMkernel adapter. Choose the TCP/IP Stack “vMotion” from Drop down as highlighted.
As you can see vMotion TCP/IP stack is selected for vmKernel adapters and it’s grayed out means you are not able to change it to select another service.
Follow the “new networking wizard” and create the vmKernel adapter.
Once created. You will see vMotion vmKernel adapter has been added by using vMotion TCP/IP stack.
Similarly add a vmKernel adapter for NFS and this time select Custom NFS stack which I have created for this purpose. Here is final configuration looks likes.
NOTE: if you already created the vmkernel adapters and want to use the TCP/IP stack against these vmKernel adapters. Then you have to delete them first and recreate it by choosing the appropriate TCP/IP stack as I did in above steps.
At that time. I was thing. It’s look pretty good and everything should work. I tried to add a NFS share.
In “New datastore” adding wizard. When I reached at this screen I got below error. Which is not so helpful by reading its description. Any one guess what this error mean? Anyone?
Its means, there is a datastore already exists with the same name which i had entered in one of the host. Of course this has nothing to do with TCP/IP stack configuration. But this is what I find during configuration so I thought I would share this as well.
Let’s try to add storage using vSphere Classic Client.
While adding I got following error. Which tells me “unable to access or connect the NFS server”.
As per the configuration which I did above. It supposed to be working. But it’s not. Let’s troubleshoot it.
After hours of digging. I reached at this point. I SSH into ESXi host and check its routes. As you can see below there is no route for vmk1 and vmk2. Which supposed to here. If you are using different network for each traffic.
Let’s see how to solve it.
The only solution which work for me to create the vmKernel with “default TCP/IP stack”. Means you should not use separate TCP/IP stack for each traffic. Which supposed to be work. But I don’t know its bug or I am using it wrong. Anyways let’s delete these vmKernel adapters and create new one with “default TCP/IP stack”.
Now again run the “add new networking” wizard as I did previous to add vmkernel adapter for vMotion and NFS. Make sure you should delete the old one first.
In wizard this time select TCP/IP Stack “default” for vMotion and select enable service “vMotion traffic”. Following the wizard and complete the steps.
Once vmKernel adapter for vMotion is created. You can verify it as shown below. It’s on default TCP/IP stack.
Similarly add vmKernel adapter for NFS traffic on default TCP/IP stack as shown below.
Once done. Let’s check the route now on ESXi host. As you can see below now all relevant routes are there as expected. Let’s try to add “NFS” storage now.
Now add storage from classic client or webclient. i find it easy to use classic client here.
Now you see it’s added successfully.
I find it very strange that TCP/IP stack configuration is not working as i suppose. logically it should work if you use separate TCP/IP stack for each traffic. But it did not. Anyhow it might be bug and fixed in later releases or i am doing it wrong.if any one find it working please let me know. But right now you see the work around of the problems. If you find it your environment. It will save your tons of time.