Monday, October 28, 2013

No SAN, No Cry

"No SAN No Cry"

No, SAN, no cry;
No, SAN , no cry;
No, SAN, no cry;
No, SAN, no cry.

'Cause - 'cause - 'cause I remember when we used to sit
In a dark server room in Trenchtown,
Oba - obaserving the bad pWWN  - yeah! -
Zoned with the good pWWN , yeah!
Storage Admins we have, oh, SAN Admins we have lost
Along the way, yeah!
In this great Nutanix converged future, you can forget your pLOGI;
So dismantle your Storage Array, I seh. Yeah!

No, SAN, no cry;
No, SAN, no cry. Eh, yeah!
Storage admin', don't shed no tears:
No, SAN, no cry. Eh!

Said - said - said I remember when we used to sit
In the dark server room in Trenchtown, yeah!
And then SAN Admin would make the Fiber Channel no shut,
I seh, PRLI crashing all through the nights, yeah!
Then SAN admin would reassign the LUN, say,
Of which to a proper  server pwwn, yeah!
Now,Your compute is your storage
And so I've got  Nutanix Converged Architecture.
Oh, while SAN is gone,
Everything's gonna be all right!
Everything's gonna be all right!
Everything's gonna be all right, yeah!
Everything's gonna be all right!
Everything's gonna be all right-a!
Everything's gonna be all right!
Everything's gonna be all right, yeah!
Everything's gonna be all right!

So no, SAN, no cry;
No, SAN, no cry.
I seh, O Server Admin - O Storage Admin', don't shed no tears;
No, SAN, no cry, eh.

No, SAN - no, SAN - no, SAN, no cry;
No, SAN, no cry.
One more time I got to say:
O Server Admin - Storage Admin', please don't shed no tears;
No, SAN, no cry.

Wednesday, October 23, 2013

vmware: Unidesk VM on NFS taking about a minute to start booting up.

Here we look at traces (NFS) to find few inefficiencies of VMware NFS client
unidesk config:

 




there is Workaround available: (from nutanix/unidesk support)

The workaround is to follow these guidelines:
1. Ensure the root of the datastore has as few directories as possible. 
2. The desktop VM's files need to be two directories below the root of the data store.

Assuming a greenfield setup, this is how to setup the directory structure.

1. Create one directory for the desktop VMs.
2. Create one directory for all the CPs and the rest of the infrastructure VMs.
3. If VMs were automatically deployed, unregister VMs, move into the appropriate subdirectory, register VMs.
4. Use the "synchronize infrastructure" in the Unidesk Management Appliance UI to discover the new VM ids which change after re-registering.

Example of a 3 desktop setup:
/vmfs/volumes/ctr1/DesktopVMs/VM1
/vmfs/volumes/ctr1/DesktopVMs/VM2
/vmfs/volumes/ctr1/DesktopVMs/VM3

/vmfs/volumes/ctr1/InfraVMs/CachePoint1
/vmfs/volumes/ctr1/InfraVMs/CachePoint2
/vmfs/volumes/ctr1/InfraVMs/CachePoint3
/vmfs/volumes/ctr1/InfraVMs/MA
/vmfs/volumes/ctr1/InfraVMs/MasterCachePoint
/vmfs/volumes/ctr1/InfraVMs/InstallationMachine1


More details on the issue:
 

How to capture NFS traffic on Nutanix, so that we can review through the wireshark.

1. Power off all the VMs other than CVM and Unidesk VM, so that we get clean trace.


sudo tcpdump -i eth1 port 2049 -w nfs.cap

 


 Bootup time in Vcenter:
 ~Unidesk-TESTVM   Completed  Administrator  pax-vCenter
start time:10/21/2013 1:53:19 PM
Bootup complete as per Vcenter: 10/21/2013 1:54:13 PM ( it still needs to boot the OS), It waits for more than
40 seconds at 95%)
 
Four things I can see that are odd :

1. NFS client is looking for entire / directory rather than focussing on the files .
(Unidesk KB specifically addresses how to reduce this issue this : http://www.unidesk.com/support/kb/unidesk-configuration-considerations-nfs-based-storage-including-nutanix-your-boot-images)


and looking  UP sre-theos2,sretheos1 which are not part of ~Unidesk directory.



2. It is creating too many vmBigTestFile ( it should be enough to create one).
3. REDO logs - looking up and then verifying and then creating the files (
so it will take while before sending create files)

repeated create and remove of redo logs ( not sure why ESXi nfs client has to do that)


4. Workers (4 workers) do not work parallely, they seem to work serially. - deduced
from wireshark and vmware.log file. During the Pause I see other worker is active.


On our system, it took about 56seconds. It is a 2400 system.
It took about 37 seconds for opening and closing of the vmdks. ( of which 13-15 seconds was ESXi waiting after open successful to close).
It is same as your system.


/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM # grep vmfs vmware.log|head -1
2013-10-21T20:28:03.795Z| vmx| I120: Command line: "/bin/vmx" "-ssched.group=host/user" "-#" "product=4;name=VMware ESX;version=5.1.0;buildnumber=1065491;licensename=VMware ESX Server;licenseversion=5.0;" "-@" "duplex=3;msgs=ui" "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM.vmx"
/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM # grep vmfs vmware.log|tail -1
2013-10-21T20:28:40.674Z| Worker#3| I120: DISKLIB-LIB   : Opened "./~Unidesk-TESTVM_13.vmdk.REDO_6uPnSb" (flags 0x8, type vmfsSparse).




/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM # grep Worker#3 vmware.log
2013-10-21T20:28:04.503Z| Worker#3| I120: DISK: OPEN scsi0:3 '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_17.vmdk' independent-nonpersistent R[]
2013-10-21T20:28:04.870Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:04.877Z| Worker#3| I120: DISKLIB-VMFS  : "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_17-flat.vmdk" : open successful (14) size = 10737418240, hd = 157193863. Type 3
2013-10-21T20:28:04.877Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_17-flat.vmdk" (0xe)
2013-10-21T20:28:04.877Z| Worker#3| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_17.vmdk' (0xe): vmfs, 20971520 sectors / 10 GB.
2013-10-21T20:28:04.922Z| Worker#3| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_17.vmdk" (flags 0xe, type vmfs).
2013-10-21T20:28:05.847Z| Worker#3| I120: DISKLIB-LIB   : DiskLibCreateCreateParam: vmfssparse grain size set to : 1
2013-10-21T20:28:06.030Z| Worker#3| I120: DISKLIB-LIB   : CREATE CHILD: "./~Unidesk-TESTVM_17.vmdk.REDO_grMSJ6" -- vmfsSparse cowGran=0 allocType=0
2013-10-21T20:28:06.033Z| Worker#3| I120: CREATE-CHILD: Creating disk backed by 'default'
2013-10-21T20:28:06.656Z| Worker#3| I120: DISKLIB-VMFS_SPARSE : VmfsSparseExtentCreate: "./~Unidesk-TESTVM_17.vmdk-delta.REDO_grMSJ6" : success
2013-10-21T20:28:06.706Z| Worker#3| I120: DISKLIB-DSCPTR: "./~Unidesk-TESTVM_17.vmdk.REDO_grMSJ6" : creation successful.
2013-10-21T20:28:07.304Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:07.310Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_17.vmdk-delta.REDO_grMSJ6" : open successful (17) size = 24576, hd = 0. Type 8

--   2 seconds

2013-10-21T20:28:09.704Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_17.vmdk-delta.REDO_grMSJ6" : closed.
2013-10-21T20:28:10.088Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:10.100Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_17.vmdk-delta.REDO_grMSJ6" : open successful (8) size = 24576, hd = 152606383. Type 8
2013-10-21T20:28:10.100Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_17.vmdk-delta.REDO_grMSJ6" (0x8)
2013-10-21T20:28:10.100Z| Worker#3| I120: DISKLIB-LINK  : Opened './~Unidesk-TESTVM_17.vmdk.REDO_grMSJ6' (0x8): vmfsSparse, 20971520 sectors / 10 GB.
2013-10-21T20:28:10.100Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1
2013-10-21T20:28:10.100Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 152606383, extentType = 0
2013-10-21T20:28:10.123Z| Worker#3| I120: DISKLIB-LIB   : Opened "./~Unidesk-TESTVM_17.vmdk.REDO_grMSJ6" (flags 0x8, type vmfsSparse).
2013-10-21T20:28:10.123Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1
2013-10-21T20:28:10.123Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 157193863, extentType = 2
2013-10-21T20:28:10.124Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 152606383, extentType = 0
2013-10-21T20:28:10.149Z| Worker#3| I120: DISK: Disk './~Unidesk-TESTVM_17.vmdk.REDO_grMSJ6' has UUID '60 00 c2 95 e9 99 ac 11-ff da ee ef 7d d0 cc 3d'
2013-10-21T20:28:10.149Z| Worker#3| I120: DISK: OPEN './~Unidesk-TESTVM_17.vmdk.REDO_grMSJ6' Geo (1305/255/63) BIOS Geo (0/0/0)


2013-10-21T20:28:10.196Z| Worker#3| I120: DISK: OPEN scsi0:8 '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_21.vmdk' independent-nonpersistent R[]
2013-10-21T20:28:10.386Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:10.495Z| Worker#3| I120: DISKLIB-VMFS  : "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_21-flat.vmdk" : open successful (14) size = 10737418240, hd = 158897845. Type 3
2013-10-21T20:28:10.495Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_21-flat.vmdk" (0xe)
2013-10-21T20:28:10.495Z| Worker#3| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_21.vmdk' (0xe): vmfs, 20971520 sectors / 10 GB.
2013-10-21T20:28:10.498Z| Worker#3| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_21.vmdk" (flags 0xe, type vmfs).
2013-10-21T20:28:11.418Z| Worker#3| I120: DISKLIB-LIB   : DiskLibCreateCreateParam: vmfssparse grain size set to : 1
2013-10-21T20:28:11.514Z| Worker#3| I120: DISKLIB-LIB   : CREATE CHILD: "./~Unidesk-TESTVM_21.vmdk.REDO_QnAph7" -- vmfsSparse cowGran=0 allocType=0
2013-10-21T20:28:11.523Z| Worker#3| I120: CREATE-CHILD: Creating disk backed by 'default'
2013-10-21T20:28:12.114Z| Worker#3| I120: DISKLIB-VMFS_SPARSE : VmfsSparseExtentCreate: "./~Unidesk-TESTVM_21.vmdk-delta.REDO_QnAph7" : success
2013-10-21T20:28:12.258Z| Worker#3| I120: DISKLIB-DSCPTR: "./~Unidesk-TESTVM_21.vmdk.REDO_QnAph7" : creation successful.
2013-10-21T20:28:13.161Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:13.168Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_21.vmdk-delta.REDO_QnAph7" : open successful (17) size = 24576, hd = 0. Type 8

 2 seconds

2013-10-21T20:28:15.207Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_21.vmdk-delta.REDO_QnAph7" : closed.
2013-10-21T20:28:15.775Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:15.819Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_21.vmdk-delta.REDO_QnAph7" : open successful (8) size = 24576, hd = 141858511. Type 8
2013-10-21T20:28:15.819Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_21.vmdk-delta.REDO_QnAph7" (0x8)
2013-10-21T20:28:15.819Z| Worker#3| I120: DISKLIB-LINK  : Opened './~Unidesk-TESTVM_21.vmdk.REDO_QnAph7' (0x8): vmfsSparse, 20971520 sectors / 10 GB.
2013-10-21T20:28:15.819Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1
2013-10-21T20:28:15.819Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 141858511, extentType = 0
2013-10-21T20:28:15.837Z| Worker#3| I120: DISKLIB-LIB   : Opened "./~Unidesk-TESTVM_21.vmdk.REDO_QnAph7" (flags 0x8, type vmfsSparse).
2013-10-21T20:28:15.838Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1
2013-10-21T20:28:15.838Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 158897845, extentType = 2
2013-10-21T20:28:15.838Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 141858511, extentType = 0
2013-10-21T20:28:15.876Z| Worker#3| I120: DISK: Disk './~Unidesk-TESTVM_21.vmdk.REDO_QnAph7' has UUID '60 00 c2 95 84 24 a5 34-e5 25 bc b7 82 0f f9 35'
2013-10-21T20:28:15.876Z| Worker#3| I120: DISK: OPEN './~Unidesk-TESTVM_21.vmdk.REDO_QnAph7' Geo (1305/255/63) BIOS Geo (0/0/0)



2013-10-21T20:28:15.957Z| Worker#3| I120: DISK: OPEN scsi0:12 '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_25.vmdk' independent-nonpersistent R[]
2013-10-21T20:28:16.235Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:16.255Z| Worker#3| I120: DISKLIB-VMFS  : "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_25-flat.vmdk" : open successful (14) size = 10737418240, hd = 154441437. Type 3
2013-10-21T20:28:16.255Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_25-flat.vmdk" (0xe)
2013-10-21T20:28:16.255Z| Worker#3| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_25.vmdk' (0xe): vmfs, 20971520 sectors / 10 GB.
2013-10-21T20:28:16.380Z| Worker#3| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_25.vmdk" (flags 0xe, type vmfs).
2013-10-21T20:28:17.299Z| Worker#3| I120: DISKLIB-LIB   : DiskLibCreateCreateParam: vmfssparse grain size set to : 1
2013-10-21T20:28:17.497Z| Worker#3| I120: DISKLIB-LIB   : CREATE CHILD: "./~Unidesk-TESTVM_25.vmdk.REDO_4Isqws" -- vmfsSparse cowGran=0 allocType=0
2013-10-21T20:28:17.500Z| Worker#3| I120: CREATE-CHILD: Creating disk backed by 'default'
2013-10-21T20:28:18.249Z| Worker#3| I120: DISKLIB-VMFS_SPARSE : VmfsSparseExtentCreate: "./~Unidesk-TESTVM_25.vmdk-delta.REDO_4Isqws" : success
2013-10-21T20:28:18.423Z| Worker#3| I120: DISKLIB-DSCPTR: "./~Unidesk-TESTVM_25.vmdk.REDO_4Isqws" : creation successful.
2013-10-21T20:28:19.280Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:19.304Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_25.vmdk-delta.REDO_4Isqws" : open successful (17) size = 24576, hd = 0. Type 8

 2 seconds

2013-10-21T20:28:21.714Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_25.vmdk-delta.REDO_4Isqws" : closed.
2013-10-21T20:28:22.184Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:22.189Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_25.vmdk-delta.REDO_4Isqws" : open successful (8) size = 24576, hd = 155686649. Type 8
2013-10-21T20:28:22.189Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_25.vmdk-delta.REDO_4Isqws" (0x8)
2013-10-21T20:28:22.189Z| Worker#3| I120: DISKLIB-LINK  : Opened './~Unidesk-TESTVM_25.vmdk.REDO_4Isqws' (0x8): vmfsSparse, 20971520 sectors / 10 GB.
2013-10-21T20:28:22.189Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1
2013-10-21T20:28:22.189Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 155686649, extentType = 0
2013-10-21T20:28:22.222Z| Worker#3| I120: DISKLIB-LIB   : Opened "./~Unidesk-TESTVM_25.vmdk.REDO_4Isqws" (flags 0x8, type vmfsSparse).
2013-10-21T20:28:22.222Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1
2013-10-21T20:28:22.222Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 154441437, extentType = 2
2013-10-21T20:28:22.222Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 155686649, extentType = 0
2013-10-21T20:28:22.253Z| Worker#3| I120: DISK: Disk './~Unidesk-TESTVM_25.vmdk.REDO_4Isqws' has UUID '60 00 c2 96 68 ce 90 d3-83 68 6b 87 bc a4 a7 f6'
2013-10-21T20:28:22.253Z| Worker#3| I120: DISK: OPEN './~Unidesk-TESTVM_25.vmdk.REDO_4Isqws' Geo (1305/255/63) BIOS Geo (0/0/0)



2013-10-21T20:28:22.297Z| Worker#3| I120: DISK: OPEN scsi1:2 '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_2.vmdk' independent-nonpersistent R[]
2013-10-21T20:28:22.868Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:23.843Z| Worker#3| I120: DISKLIB-VMFS  : "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_2-flat.vmdk" : open successful (14) size = 10737418240, hd = 151295754. Type 3
2013-10-21T20:28:23.843Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_2-flat.vmdk" (0xe)
2013-10-21T20:28:23.843Z| Worker#3| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_2.vmdk' (0xe): vmfs, 20971520 sectors / 10 GB.
2013-10-21T20:28:23.907Z| Worker#3| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_2.vmdk" (flags 0xe, type vmfs).
2013-10-21T20:28:24.402Z| Worker#3| I120: DISKLIB-LIB   : DiskLibCreateCreateParam: vmfssparse grain size set to : 1
2013-10-21T20:28:24.483Z| Worker#3| I120: DISKLIB-LIB   : CREATE CHILD: "./~Unidesk-TESTVM_2.vmdk.REDO_yXB1Cf" -- vmfsSparse cowGran=0 allocType=0
2013-10-21T20:28:24.532Z| Worker#3| I120: CREATE-CHILD: Creating disk backed by 'default'
2013-10-21T20:28:25.310Z| Worker#3| I120: DISKLIB-VMFS_SPARSE : VmfsSparseExtentCreate: "./~Unidesk-TESTVM_2.vmdk-delta.REDO_yXB1Cf" : success
2013-10-21T20:28:25.352Z| Worker#3| I120: DISKLIB-DSCPTR: "./~Unidesk-TESTVM_2.vmdk.REDO_yXB1Cf" : creation successful.
2013-10-21T20:28:26.020Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:26.026Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_2.vmdk-delta.REDO_yXB1Cf" : open successful (17) size = 24576, hd = 0. Type 8

 2 seconds
2013-10-21T20:28:28.685Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_2.vmdk-delta.REDO_yXB1Cf" : closed.
2013-10-21T20:28:29.066Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:29.097Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_2.vmdk-delta.REDO_yXB1Cf" : open successful (8) size = 24576, hd = 140744474. Type 8
2013-10-21T20:28:29.097Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_2.vmdk-delta.REDO_yXB1Cf" (0x8)
2013-10-21T20:28:29.097Z| Worker#3| I120: DISKLIB-LINK  : Opened './~Unidesk-TESTVM_2.vmdk.REDO_yXB1Cf' (0x8): vmfsSparse, 20971520 sectors / 10 GB.
2013-10-21T20:28:29.097Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1
2013-10-21T20:28:29.097Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 140744474, extentType = 0
2013-10-21T20:28:29.116Z| Worker#3| I120: DISKLIB-LIB   : Opened "./~Unidesk-TESTVM_2.vmdk.REDO_yXB1Cf" (flags 0x8, type vmfsSparse).
2013-10-21T20:28:29.116Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1
2013-10-21T20:28:29.116Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 151295754, extentType = 2
2013-10-21T20:28:29.116Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 140744474, extentType = 0
2013-10-21T20:28:29.139Z| Worker#3| I120: DISK: Disk './~Unidesk-TESTVM_2.vmdk.REDO_yXB1Cf' has UUID '60 00 c2 9a 23 51 be 1d-8c 5a 0e c8 72 31 85 d2'
2013-10-21T20:28:29.139Z| Worker#3| I120: DISK: OPEN './~Unidesk-TESTVM_2.vmdk.REDO_yXB1Cf' Geo (1305/255/63) BIOS Geo (0/0/0)
2013-10-21T20:28:29.235Z| Worker#3| I120: DISK: OPEN scsi1:6 '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_6.vmdk' independent-nonpersistent R[]
2013-10-21T20:28:29.644Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:29.651Z| Worker#3| I120: DISKLIB-VMFS  : "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_6-flat.vmdk" : open successful (14) size = 10737418240, hd = 199005995. Type 3
2013-10-21T20:28:29.651Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_6-flat.vmdk" (0xe)
2013-10-21T20:28:29.651Z| Worker#3| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_6.vmdk' (0xe): vmfs, 20971520 sectors / 10 GB.
2013-10-21T20:28:29.656Z| Worker#3| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_6.vmdk" (flags 0xe, type vmfs).
2013-10-21T20:28:30.684Z| Worker#3| I120: DISKLIB-LIB   : DiskLibCreateCreateParam: vmfssparse grain size set to : 1
2013-10-21T20:28:30.766Z| Worker#3| I120: DISKLIB-LIB   : CREATE CHILD: "./~Unidesk-TESTVM_6.vmdk.REDO_oUoy4b" -- vmfsSparse cowGran=0 allocType=0
2013-10-21T20:28:30.777Z| Worker#3| I120: CREATE-CHILD: Creating disk backed by 'default'
2013-10-21T20:28:31.230Z| Worker#3| I120: DISKLIB-VMFS_SPARSE : VmfsSparseExtentCreate: "./~Unidesk-TESTVM_6.vmdk-delta.REDO_oUoy4b" : success
2013-10-21T20:28:31.423Z| Worker#3| I120: DISKLIB-DSCPTR: "./~Unidesk-TESTVM_6.vmdk.REDO_oUoy4b" : creation successful.
2013-10-21T20:28:32.308Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:32.341Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_6.vmdk-delta.REDO_oUoy4b" : open successful (17) size = 24576, hd = 0. Type 8

 --- 2 seconds 
2013-10-21T20:28:34.457Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_6.vmdk-delta.REDO_oUoy4b" : closed.
2013-10-21T20:28:34.717Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:34.722Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_6.vmdk-delta.REDO_oUoy4b" : open successful (8) size = 24576, hd = 181901123. Type 8
2013-10-21T20:28:34.722Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_6.vmdk-delta.REDO_oUoy4b" (0x8)
2013-10-21T20:28:34.722Z| Worker#3| I120: DISKLIB-LINK  : Opened './~Unidesk-TESTVM_6.vmdk.REDO_oUoy4b' (0x8): vmfsSparse, 20971520 sectors / 10 GB.
2013-10-21T20:28:34.722Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1
2013-10-21T20:28:34.722Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 181901123, extentType = 0
2013-10-21T20:28:34.769Z| Worker#3| I120: DISKLIB-LIB   : Opened "./~Unidesk-TESTVM_6.vmdk.REDO_oUoy4b" (flags 0x8, type vmfsSparse).
2013-10-21T20:28:34.769Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1
2013-10-21T20:28:34.769Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 199005995, extentType = 2
2013-10-21T20:28:34.769Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 181901123, extentType = 0
2013-10-21T20:28:34.805Z| Worker#3| I120: DISK: Disk './~Unidesk-TESTVM_6.vmdk.REDO_oUoy4b' has UUID '60 00 c2 9c 9e 2d 8c 87-b7 d5 50 37 2a 7d 39 e7'
2013-10-21T20:28:34.805Z| Worker#3| I120: DISK: OPEN './~Unidesk-TESTVM_6.vmdk.REDO_oUoy4b' Geo (1305/255/63) BIOS Geo (0/0/0)


2013-10-21T20:28:34.897Z| Worker#3| I120: DISK: OPEN scsi1:14 '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_13.vmdk' independent-nonpersistent R[]
2013-10-21T20:28:35.334Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:35.348Z| Worker#3| I120: DISKLIB-VMFS  : "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_13-flat.vmdk" : open successful (14) size = 3758096384, hd = 146904906. Type 3
2013-10-21T20:28:35.348Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_13-flat.vmdk" (0xe)
2013-10-21T20:28:35.348Z| Worker#3| I120: DISKLIB-LINK  : Opened '/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_13.vmdk' (0xe): vmfs, 7340032 sectors / 3.5 GB.
2013-10-21T20:28:35.438Z| Worker#3| I120: DISKLIB-LIB   : Opened "/vmfs/volumes/2fbb0409-ee3410bd/~Unidesk-TESTVM/~Unidesk-TESTVM_13.vmdk" (flags 0xe, type vmfs).
2013-10-21T20:28:35.876Z| Worker#3| I120: DISKLIB-LIB   : DiskLibCreateCreateParam: vmfssparse grain size set to : 1
2013-10-21T20:28:35.954Z| Worker#3| I120: DISKLIB-LIB   : CREATE CHILD: "./~Unidesk-TESTVM_13.vmdk.REDO_6uPnSb" -- vmfsSparse cowGran=0 allocType=0
2013-10-21T20:28:36.002Z| Worker#3| I120: CREATE-CHILD: Creating disk backed by 'default'
2013-10-21T20:28:36.632Z| Worker#3| I120: DISKLIB-VMFS_SPARSE : VmfsSparseExtentCreate: "./~Unidesk-TESTVM_13.vmdk-delta.REDO_6uPnSb" : success
2013-10-21T20:28:36.780Z| Worker#3| I120: DISKLIB-DSCPTR: "./~Unidesk-TESTVM_13.vmdk.REDO_6uPnSb" : creation successful.
2013-10-21T20:28:37.618Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:37.655Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_13.vmdk-delta.REDO_6uPnSb" : open successful (17) size = 12288, hd = 0. Type 8

3 seconds

2013-10-21T20:28:40.221Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_13.vmdk-delta.REDO_6uPnSb" : closed.
2013-10-21T20:28:40.562Z| Worker#3| I120: nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-10-21T20:28:40.594Z| Worker#3| I120: DISKLIB-VMFS  : "./~Unidesk-TESTVM_13.vmdk-delta.REDO_6uPnSb" : open successful (8) size = 12288, hd = 157652962. Type 8
2013-10-21T20:28:40.594Z| Worker#3| I120: DISKLIB-DSCPTR: Opened [0]: "~Unidesk-TESTVM_13.vmdk-delta.REDO_6uPnSb" (0x8)
2013-10-21T20:28:40.594Z| Worker#3| I120: DISKLIB-LINK  : Opened './~Unidesk-TESTVM_13.vmdk.REDO_6uPnSb' (0x8): vmfsSparse, 7340032 sectors / 3.5 GB.
2013-10-21T20:28:40.594Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 1, numSubChains = 1
2013-10-21T20:28:40.594Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 157652962, extentType = 0
2013-10-21T20:28:40.674Z| Worker#3| I120: DISKLIB-LIB   : Opened "./~Unidesk-TESTVM_13.vmdk.REDO_6uPnSb" (flags 0x8, type vmfsSparse).
2013-10-21T20:28:40.674Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain: numLinks = 2, numSubChains = 1
2013-10-21T20:28:40.674Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 146904906, extentType = 2
2013-10-21T20:28:40.674Z| Worker#3| I120: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 157652962, extentType = 0
2013-10-21T20:28:40.757Z| Worker#3| I120: DISK: Disk './~Unidesk-TESTVM_13.vmdk.REDO_6uPnSb' has UUID '60 00 c2 96 78 0e 37 cd-a2 10 ae b7 8f 64 f0 68'
2013-10-21T20:28:40.757Z| Worker#3| I120: DISK: OPEN './~Unidesk-TESTVM_13.vmdk.REDO_6uPnSb' Geo (456/255/63) BIOS Geo (0/0/0)