Following on from our June 2013 Citrix Support Secrets webinar which covered Top Troubleshooting Tips and Techniques for Citrix XenServer Deployments, there was a series of questions asked during the session that we didn’t get to answer. See the questions and answers below. BTW if you missed the webinar and would like to watch it on-demand, you can find it here.
Q: Can we have a brief overview on the new licensing on 6.2 please
Q: Does chains create when we copy or move VMs?
If it’s a full copy no. If you are doing a Storage XenMotion then snapshots are used for it… and you will have a chain.
Q: Do you have a solution for recreate chain and free space on the live VMs(such as database etc) without downtime ?
The coalesce of the snapshop chain doesn’t require downtime.
Q: How can we clean disk after snapshots?
You can remove the snapshots from XenCenter and it will coalesce the chain… if you reach the limit of 30 snapshot coalesce might not work anymore and you will need to full copy the VM to consolidate the chain in one single VHD.
Q: I have a small problem regarding a VM that crashed. What hapenned is: I tried to do a force shutdown and reboot on the VM in XenCenter but it didn’t work. I went into command line and tried a reboot –force but still no luck. So I went ahead and copied the vm uuid, I did a list_domains, and copied the domain ID. I then did a destroy_domain command, then a shutdown –force. It worked! So I went ahead and started the VM, but then I had this dreaded VDI not available error. So i copied the vm’s VDI uuid, I did a vdi-forget of the VDI, rescanned the SR, reattached the VDI to the vm, and now I can boot again! But now, I don’t know why, but the VDI is attached to my “troubled vm” and also to a “Control domain on host: ***”… and I just don’t know how to detach that VDI from this control domain thing, because I need to move the vm on another SR, and when I do this, I get an error saying that there is already a disk at position 0. Any idea?
The reason you could not start it was because the lock was not cleared but when you did the copy the VDI got attached to dom0, you could try to remove the VBD to Dom0 but the easiest way unfortunately would be to reboot the host
Q: How can you detect failed local hard disk drives on a xenserver since it doesn’t show an alert on xencenter?
You will see I/O errors on /var/log/messages and /var/log/kern.log, and very likely on the physical console of the server.
Q: I have a problem with install a new driver for intel 10Gb nic drivers.. if I remove modules a few seconds later the same module was insmod… I must run xenserver in a fail-safe mode.. on citrix site I was only information about rmmod old module… nothing about run server in safe mode.. servers doesn’t have any VMs.. any advice?
If you need to run the server in safe mode simply aafter a reboot as soon as you get a prompt type menu.c32 and then select safe-mode… On the other hand installing a new driver should not be a problem as it will be loaded after the reboot anyway, so I’m not entirely sure what is going on in your environment…
Q: I have situation where C drive full and XenServer was setup build VMs on C drive.
I would like export or backup those VMs and move to another XenServer. What is best way to export Virtual machines from XenServer or how to backup to external Hard drive.
When the export you need to shutdown the VM and then do right click on it “export…” you will need to give it a destination with enough free space, the VM will be stored in a file that can be imported once again from XenCenter – File- Import to the desired SR. Another option would be to copy the VM to a new SR, once again with the VM shutdown right click on it and select copy, and paste it on a SR with enough free space. If you use XS 6.1 you could also do a Storage Xenmotion and move the VM to a SR with enough free space with no downtime involved.
Q: I imported a linux VM from VMWare to XenServer 6.1. I intalled xentools into the VM, but XenCenter keeps saying “Xentools not installed”, what should I check?
Make sure that the VMware tools are completely removed first, then removed the XenTools and install them again.
Q: In future versions will we still need to over-provision the SR to allow enough space for the snapshots?
AFAIK yes, there’s no way of
Q: “Dont use snapshots as backup” – what about products like PHDVirtual etc with guest tools?
You can use the snapshot as a tool to make the backup, take the snapshot — export the vm – remove the snapshot… the idea is to not consider the snapshot a backup by itself.
Q: I use a low-level DD command to reclaim space on thin-provisioned SRs using block storage, (lvcreate on VG, DD to use 100% of unallocated disk, lvremove to delete empy disk). Can you comment? Have you seen others use this?
A: Brave man! I haven’t seen this done by any other customer before, sounds kind of dangerous if not extremely careful.
Q: Can GC be automated, or manually scheduled?
GC happens automatically… you can force it, there’s a CTX article about it
Q: What happens under the hood with StorageVMotion wrt VHD disks? Does it use snapshots, or transfer VM?
Watch this: http://support.citrix.com/article/CTX135362
Q: Is there a way to have snapshot be done to another storage repository other than where the VM resides if the current volume does not have enough temp space for the snap?
Unfortunately not… been requested many times before, ask your Sale guys to request it as if there’s enough people asking for it then there will be a strong business case to work on it.
Q: Is we have to do any twicks when we are upgrading to version to 6.0.2?
Not really, advice would be to go to 5.6 SP 2 first if you are using an older version of XS.
Q: It’s possible to merge VHDs? for example one from a snapshot and his VHD parent?
This is call coalesce and it happens when you remove the snapshot.
Q: With the introduction of XenServer 6.2 Citrix has announced that it’s planning to drop iSL.
Q: What is the preferred way to use an external SR – let’s say NetApp FAS2220?
Not sure what you mean. Looks like to want to know if to use NFS or fiber/iSCSI, right? Is hard to answer, fiber/iSCSI should provide better performance but with NFS you could have thin provisioning… So is more of a matter of preference and what you intend to run.
Q: On a XenServer 5.6 system, can you add drives to increase the disk space on a hardware based HBA SAN (restriping on the hardware) while VMs are running live, or must all VMs be shut down first? Not iSCSI, but fiber to a SAN.
I haven’t tested myself… but I reckon that if you are able to increase the size of the LUN without breaking it we should be able to increase the size of the SR without affecting the VMs… once again, this is just theory I would test it first… To further add to the previous, at some point you will need to shutdown the VM because you will need to unplug/replug the SR in order of the new free space to be recongnized.
Q: Hi, have done an upgrade of 14 XS 5.6 FP1 Hosts to XS 6.0.2 including 5 hotfixes. The complete upgrade took about 17 hours. Is this the normal lenght for an upgrade ? I was using the Rolling Upgrade and Install Software update from within XenCenter. Is there a way to speed the whole process up ? See the question in the forum http://forums.citrix.com/thread.jspa?messageID=1744792�
Many HF require reboots. and VMs need to be migrated across as to leave the host with no VM, this takes a lot of time, I reckon that is what happened to you… you can spped up the process by appliying HF in bulk and reducing the amount of reboot and by shutting down unnecesary VMs as to reduce the need to migration… unfortunately this will be a manual process
Q: Why is xenserver, 6.1 expecially, so bad with the coalesce issue? Its very annoying having to try and keep the LUN’s clean by manually having scan and run the coalesce commands plus suspend VM/shutdown etc. Also the rogue VHD’s that appear that aren’t linked to a VM and show up as used space.
Sorry to hear you are having problem with the coalesce, are you using PHD maybe? I know there have been some issue sreported by users of PHD that why I asked. Unfortunately I don’t have a silver bullet to fix your problem , we will need to look at in detail.
Q: Had an issue last night when we rebooted our FW cluster which the xen hosts route through. During this two of our hosts rebooted when losing access to NFS HA SR, 3 didn’t? This caused no ends of trouble, why would this happen? Very poor. Uploaded the crash debug to citrix auto support and it said there were no issues.
The way HA works indicates that if access is lost the state file (which I reckon you lost according to the description) then an election happens to figure out which sever are no longer responding in order to fence them to prevent a split brain scenario, the pool is divided according to what they saw last and those in minority are fenced. This is just a glance as the way it works, as is more complex that than…
Q: Why is it that LUNS don’t plug sometimes after reboot…have to do a iscsiadm -m node -L all.
Without having a look is hard to answer, but could it be that the login timeouts after the reboot?
Q: Why does xencenter tell us that our licenses will expire soon when they are all set to never? Very annoying.
That looks like a bug, please install the latest version of XenCenter and see if the problem remains…
Q: Why do you have to diable the HA cluster to do so many tasks with hosts/VMs?
Because we need to prevent a fence and because HA needs to calculate if enough resources are available for protected VMs, among other things.
Q: Under netapp-sr, how do snapshot handling work? how do snapshot in general co-op with live xen storage migration? how does storage link co-op with live storage xen migration?
See http://support.citrix.com/article/CTX137828 , specifically: 18.104.22.168. Creating Integrated StorageLink SRs for NetApp
Q: Unfortunately we have manually removed vdi’s. what is the best way to fix this?
The only option is rto recover from backup
Q: We have 24 Hosts in a resource Pool and more to add in the future. The admin guide talks about 16 max. Is this going to cause a problem?
Having more that 16 hosts in a pool is not advisable and it not supported, it might work but you might starve the master of resources.
Q: We have had problems in XS 6.1 were NIC suddenly are no longer been recognized.. We were told that this was a known issue by XS 6.1 and that it will be a good idea to downgrade to XS 6. Is this true, have Citrix posted a fix for this problem? I have also seen some blogs about the release of XS 6.2 not sure is this version does have fixed some of the problems than his predecesor.
Unfortunately you are giving us enough details to answer your question (model and brand of the NIC for example), if you were advice to run an earlier version of Xs I reckon it could be that the hardware is not certified to run on the newer versions perhaps? Please check if your hardware is supported here: http://hcl.xensource.com/
Q: We tried to manually reclaim disk space on deleted xenserver snapshots, but it did not work. We were told by Citrix to copy vm onto a usb device and then import vm and that would bring back the 20 gigabytes of space we are missing. Is this the only solution?
That is a solution because it will consolidate the snapshot chain into one single VHD
Q: What the best pratice to upgrade XS, just shutdown VM and upgrade ?
Good practice indicates that you should take a backup before making any changes, however the upgrade is pretty much straight forward, please see: http://docs.vmd.citrix.com/XenServer/6.1.0/1.0/en_gb/installation.html#upgrading
Q: When i try to snapshot a particular VDI. I get cannot create Snapshot of a Snapshot VDI, operation unsupported. The VM had previously crashed and the tech had imported the vm from a previous snapshot
Please make a full copy of the VM, on the new copy you should be able to snapshot as usual…
Q: While migrating from one pool to a new storage pool I shutdown a VM export it from current pool, import to new pool and get VDI not available. I’ve worked with Xencenter support but cannot get an answer.
If you check in /var/log/SMlog immediately after you get the error you should be able to see in details what is going on with the VDI.
Q: Will hotfixes for Xen 6.2 be applied through Xen Center or will they have to be applied via the CLI?
You can already apply hotfixes from XenCenter, that will not change in 6.2/
Q: xencenter 6.1 hangs on me often. restarting or dieconnecting/reconnecting pool usually fixes
Does it happens on any other PC in your environment? We need to verify if it’s a problem with your PC or with XC itself… if it doesn’t happens to anyone else in your environment then try to install the latest version of XC and verify if the problem remains…
The July 2013 webinar will cover: Troubleshooting XenApp with the Citrix Diagnostic Toolkit.
When problems occur, support engineers need data points, debug tracing and context information to help determine root causes. Preparation and organization of commonly used tools has always been a time-consuming challenge, especially during outages. The Citrix diagnostics toolkit (CDT) addresses these challenges by rapidly deploying a suite of tools and options in an easy-to-use structured format.
During this session you will learn:
• What is the Citrix Diagnostics Toolkit?
• How and when to use the CDT?
• How the CDT helps Citrix deliver better technical support?
When: Thursday, 25th July 2013 @ 12pm EST/6pm CET
How: REGISTER NOW!
Looking forward to seeing many of you online.