(UPDATE: There is an APPENDIX at the bottom added for troubleshooting patch application issues.  These can also be read here.)

Hello fellow Citrites, Clients, Technology Partners and Community Members!

 

I hope each of you found some time over the Holiday Season to spend time with friends, family, loved ones and your XenServer.  I know I did!  I had planned to blog more, but I was busy testing XenServer 6.2 SP 1 in all sorts of configurations, upgrades and VM testing: all with a big smile on my face.  I’m just “bloggin’” to share practices, information and with that said:

Let this begin this first of many articles – from my Virtual Desktop to you (literally and virtually, but never mind that for now).  These articles are are meant to apply mental steroids to the “Internets’ Webs’” to enhance metabolizing information, build institutional knowledge at an exponential rate and live with the side effects of success!

The primary focus of this article is to emphasize steps for successful application of hotfixes or services packs.  With the release of XS62ESP1 (Service Pack 1) for XenServer 6.2, this article will lean upon the latest release as a point of reference, but rest assured that the steps one should take, adopt and turn into practice (along with best practices) will be discussed.

 

So, without further ado, Blogs @ Citrix.com presents : “From My Virtual Desktop: Applying a Hotfix or Service Pack” …

 

 

Step 1. It is your environment…

Yes.  Step 1 is to remind myself and you, the reader, that this is your environment.  While we test and test and test our products at Citrix, I want everyone to take a step back and think about your environment outside of just “Citrix products”.  When faced with rolling out updates from solid-state equipment to software patches for your end-users, just ask yourself:

 

  • “How much of my environment is in production?”
  • “Can I afford down time for the maintenance?”
  • “Have I read over the release notes and supplemental guides to assist with the roll out of hotfixes, service packs or other software?”
  • “Am I required to update anything else, such as device firmware, system BIOS or apply other hotfixes first?”
  • “Have I checked compatibility lists, known issues, limitations, issues-addressed or vendor-based, community forums for other user-related experiences?”
  • “Do I have any questions for support or of the vendor’s community?”
  • Do I have a test machine or test environment?

 

These are just a few questions that I have learned to roll around in my head before I plan to apply any type of software, hardware or solid-state firmware in my professional, home and customer environments: especially if I have a test machine or environment to work with.

If I am unsure about any aspect I always contact my vendor’s technical support.  Now, focusing back on Citrix and XenServer: if you are ever unsure, please reach out to Citrix Support or our Citrix Discussion Forums.

 

 

Step 2. XenServer and… a Razor?

I am a rather huge fan of Occam and his Razor.  If you do not know who he is or how sharp his razor actually is, check out the following link or just let me sum it up:

 

  • “The simplest answer is usually the correct one.”

 

I know, right?  He is THAT GOOD.  Though, since he isn’t alive I am going to discuss what I do in my own personal time or at work when it comes to applying a hotfix, service pack or even an upgrade.

 

 

Step 3.  “Backup Stuff”

Yup.  Disks, USB drives and external drives are a dime-a-dozen.  In the time spent to make a call asking for support to recover data, well, you could have already backed up mission critical data.  Note: that is not a verbal jab.  Remember restore points in Windows ME?  Yeah, I neglected to use those as well until…

The point is that sometimes things can just go wrong.  This is why infrastructure to data security, backup, redundancy and complete disaster recovery procedures are part of the CISSP exams as well as… well, a large subset of normal business operational procedures.  So before approaching the rollout of Citrix updates (or any product update) ensure that all assets are tagged, classified and backed up accordingly.

I urge a stop at http://support.citrix.com as more often than not, the documentation is directly there regarding this type of information: release notes, installation guides and so forth.  One example is this gem: http://support.citrix.com/article/CTX121282 (among many more KB Articles, Best Practices and Product-centric notes) for backup, recovery and tools to assist with such tasks.  I know, I know… but even “back in the day” we pointed this out: http://docs.vmd.citrix.com/XenServer/4.0.1/installation/apbs04.html.  Cool, eh?

We do have a wealth of CTX/Knowledge Base articles, but if those do not answer your questions please call Citrix Support or reach out to our Citrix Discussion Forums.  Understanding what type of data, extent of data to backup, format (or even data that can be just off-lined) before rolloing out software is a “Top 3 Best Practice” or “Best Way To Avoidable A Problem”.

If you have the ability and want to lean towards the side of absolute complete caution:

 

  • Fully export all VMs to a separate, redundant data source
  • Export metadata and – if you have a Pool – export the Pool’s database to a redundant, data source
  • OR… just clone your entire system onto test equipment and apply any updates there

 

I’ve made the gamble before with various types of software/hardware solutions long ago and came out a winner, but I shouldn’t have risked it.  A fly gets sucked into a machine, a rogue pack of squirrels chew their way through the town’s power grid.  Or worse!  Someone’s $6.00 cup of coffee turns all my hard, non-backed up work into a fiscal nightmare!

 

 

Step 4.  Always read the release notes, what is covered and ask questions where questions arise

This is not a polite suggestion or recommendation just for our wonderful Citrix clients, partners and community.  Oh, no – far from it!  This should be a muscle-memory behavior for anyone using a product/solution by any vendor for any purpose in their environment that costs them (or their employers).

Besides… if you read the previous section – “Step 3. Backup Stuff” – you are now ahead of most people performing the same roll out of software!

Here is a non-computing example of what I mean:

 

  • I buy a case of Soda Pop for the office
  • I see the nutrition contents, albeit barely readable, as the case is all tattered and torn
  • I can see that the cans don’t look right: warped, dented and one is leaky
  • They have a strange layer of dust atop the can
  • I see part of an expiration date, but not all of it
  • It’s on sale, but there are no clear instructions as to “Since it was on the shelf, it is okay to buy” or “If the cans are warped, dented and one is leaky do not use”

 

The questions “Should I wash these and drink?” or “Eh – it is soda.  Doesn’t phosphoric acid kill germs?” should never lead the consumption of such Soda Pop without contacting the vendor.  “Bob’s can’t do his job for a while?”  Why?  “He had a soda pop and says he has temporarily lost all vision.  Should be okay in a week, though.”

Oh.  No…

As you read the release notes, if you find yourself not being described as the “Target Audience” or if other per-requisites are not giving you the warm and fuzzy feelings, then please call Support or raise a question within the Citrix Discussion Forums.  Also, if there are hardware, firmware or other updates required to your systems – especially for upgrades – contact your hardware vendor(s) as to ensure your solid-state systems are up-to-date.  Such examples are related to, but not limited to:

 

  • Hardware is supported via the XenSource.org Hardware Compatibility List (especially for major revision upgrades or installs)
  • BIOS Revision is current
  • BIOS Power/Virtualization settings are correct
  • Card-based or other solid-state firmware is at a supported revision/version
  • Read for any deprecated features or functions within the release notes, such as (for XenServer 6.2.0 http://support.citrix.com/article/CTX137826)

 

 

Step 5.  Disk Space and other Checks

All of the aforementioned taken into account, before I just go off and install “things”, I check my environment before I move forward.  The following will make reference to XenServer 6.2 Service Pack 1, but the concepts can be applied to hotfixes and upgrades (alike).

 

My environment isn’t broke, so should I “fix it”?

  • This is an important question, specifically for Citrix Support and requirements for support
  • If we have security-related releases, yes — apply them
  • If your product is about to be end-of-life, yes — apply it to keep support current
  • If all else fails or there are no answers, again, call Support or reach out to the Citrix Discussion Forums

 

Is my environment compatible with this hotfix or service pack?

  • I know, the dreaded HCL – mentioned twice now
  • I still double-check my hardware against this to rule out any items that may render me “support-less” or incompatible with the software I want to roll out
  • If I am still in-doubt – again, not just for Citrix – I contact Support or reach out to vendor forums.

 

How many target systems do I need to apply this hotfix/service pack to?

  • Short answer: if all research has been done, all applicable servers
  • If a new version of XenCenter is available, this must be upgraded first
  • A numbers game, but knowing this applies to your strategy (as an administrator) in applying a hotfix or service pack
  • If you have stand-alone hosts, you can take a staged approach as the state-db.xml file is not shared across hosts
  • If you have a Pool (or Pools) you must apply the software to all hosts in all Pool(s) – starting with the Pool Master
  • If a new version of XenTools is bundled you will need to update tools for each VM after applying the hotfix or service pack

 

If I download the ZIP file, can I open it up to see the “XS62ESP1.update” file or do I get an error?

  • TCP/IP has come along way, but I always check my downloads for integrity – especially if it is ZIP format
  • If one cannot open the ZIP file, then you probably need to re-download the software or contact Support

 

Did I eject all mounted ISO images from VMs?

  • All iso images attached to VMs should be ejected before applying the patch
  • Do do this, run the following command from the master (if no CD image is mounted, it will say so… expect much output):

 

 xe vm-cd-eject --multiple

 

How much disk space do I have available on these target systems (because the service pack/hotfix is going to need space to install)?

  • This is why I download my patches first and look at the size of the XSxxxxx.update files as these will be further decompressed when applying to a host
  • If XSxxxxx.update is 300MiB, I make sure I have 2-3.5 times that amount of space available on each my target host (just in-case)
  • If you run out of disk space the install will fail

 

You can check available disk space using the following command on your XenServer host from XenCenter or via an SSH client, such as “putty” (example output provided, too):

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1  4.0G 2.4G 1.4G  64% /
none       1.1G 4.0K 1.1G  1%  /dev/shm
/opt/xensource/packages/iso/XenCenter.iso
           52M  52M  0     100% /var/xen/xc-install

 

From the example provided, I should have enough space.  If not, I could remove old patch files from /var/patch and older log files from /var/log as follows:

 

cd /var/log
rm -rf *gz
cd /var/patch
rm -rf *

 

I have questions… who do I talk to?

  • That is what Citrix Support and the Citrix Discussion Forums are for
  • Never hesitate to call us: especially if you have support!

 

 

Step 6.  How to apply a patch to a stand-alone host…

Much of this information may be repetitive, but it is written as such for a purpose.  Through repetition comes knowledge.  From applied knowledge comes experience.  If I can impart my knowledge to assist in your experience with XenServer as an healthy one, then I am doing what I love (aka, My Job)!

As with any update (hotfix or service pack) that is released for an existing or new version of XenServer, download and upgrade all XenCenter clients first!

Let me repeat that:

As with any update (hotfix or service pack) that is released for an existing or new version of XenServer, download and upgrade all XenCenter clients first!

XenCenter can be used to facilitate the application of a hotfix or service pack, but you have to have a licensed copy to do this.  From my own experience and desire for control, I prefer to install patches from the command line.  The choice is yours, but at least run through the checks discussed here for disk space.

I start with the downloading of a hotfix or service pack after updating XenCenter (if required, such as it is to apply XenServer 6.2 SP1 successfully).  Decompressing the ZIP file, I can be assured that XSxxxxxx.update file is in-tact.  I then check my disk space on my root (sda) partition:

 

cd /
df -h  <-- "disk filesystem, human readable"
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 4.0G 2.4G 1.4G 63%    /   <--- I have 1.4G available, so I am good albeit close!!
none 1.1G 4.0K 1.1G 1% /dev/shm /opt/xensource/packages/iso/XenCenter.iso

I clean out any “older” patch files from /var/patch and any “older” logs from /var/log on my host:

 

cd /var/log
rm -rf *gz
cd /var/patch
rm -rf *

I then secure copy the XS62ESP1.update file (using scp from a Linux host or WinSCP from a Windows host) to root’s home directory of my stand-alone XenServer:

 

scp XS62ESP1.update root@myxenserverhost:/root/

 

Now I can start to apply the hotfix/service pack.  I can’t just jump in and install it as I need to make XAPI aware of the hotfix/service pack:

 

xe patch-upload filename=XS62ESP1.update
0850b186-4d47-11e3-a720-001b2151a503

 

It will spit out a UUID — 0850b186-4d47-11e3-a720-001b2151a503 –copy and paste this as you will need it (along with your stand-alone server’s UUID):

 

xe host-list <-- I execute this
uuid ( RO) : 17aee7b6-9f36-4c97-8616-05799a5b303e <-- My host UUID
 name-label ( RW): myxenserverhost
 name-description ( RW): Default install of XenServer

 

Now that XAPI is aware of this hotfix/service pack I can then install it by using the “xe patch-apply” command along with the UUID of the hotfix/service pack that was displayed before and the UUID of my stand-alone host:

 

xe patch-apply uuid=0850b186-4d47-11e3-a720-001b2151a503 host-uuid=17aee7b6-9f36-4c97-8616-05799a5b303e

 

This is a rather large patch, so wait and eventually you will see:

 

Preparing... ##################################################
 xen-device-model ##################################################
 Preparing... ##################################################
 xen-hypervisor ##################################################
 Preparing... ##################################################
 xen-tools ##################################################
 Preparing... ##################################################
 xen-firmware ##################################################
 Preparing... ##################################################
 blktap ##################################################
 Preparing... ##################################################
 sm ##################################################
 Preparing... ##################################################
 md3000-rdac-modules-xen-2.6.##################################################
 Preparing... ##################################################
 md3000-rdac-modules-kdump-2.##################################################
 Preparing... ##################################################
 kernel-xen ##################################################
 Preparing... ##################################################
 kernel-kdump ##################################################
 openvswitch-modules-xen-2.6.##################################################
 Preparing... ##################################################
 openvswitch-modules-kdump-2.##################################################
 Regenerating initrds for kernels
 Preparing... ##################################################
 xapi-core ##################################################
 Preparing... ##################################################
 xapi-xenopsd ##################################################
 Preparing... ##################################################
 xapi-networkd ##################################################
 Preparing... ##################################################
 elasticsyslog ##################################################
 Shutting down kernel logger: [ OK ]
 Shutting down system logger: [ OK ]
 Starting system logger: [ OK ]
 Starting kernel logger: [ OK ]
 Shutting down kernel logger: [ OK ]
 Shutting down system logger: [ OK ]
 Starting system logger: [ OK ]
 Starting kernel logger: [ OK ]
 Preparing... ##################################################
 hwdata ##################################################
 Preparing... ##################################################
 xapi-xe ##################################################
 Preparing... ##################################################
 perf-tools-rrdd-gpumon ##################################################
 Starting XCP RRDD plugin xcp-rrdd-gpumon: [ OK ]
 Preparing... ##################################################
 Stopping XCP RRDD plugin xcp-rrdd-iostat: [ OK ]
 Stopping XCP RRDD plugin xcp-rrdd-squeezed: [ OK ]
 Stopping XCP RRDD plugin xcp-rrdd-xenpm: [ OK ]
 perf-tools-rrdd-plugins ##################################################
 Starting XCP RRDD plugin xcp-rrdd-iostat: [ OK ]
 Starting XCP RRDD plugin xcp-rrdd-squeezed: [ OK ]
 Starting XCP RRDD plugin xcp-rrdd-xenpm: [ OK ]
 Preparing... ##################################################
 v6d ##################################################
 Preparing... ##################################################
 vgpu ##################################################
 Latest '/opt/xensource/packages/iso/xs-tools-6.2.0-3.iso'
 9bdefc9e-84b0-27e8-2a65-978bd6ff4e4e is the local tools SR: scanning
 Done

 

With the “Done” text presented I can either shutdown my VMs one-at-a-time and execute a “reboot” command for my stand-alone server.

A quick route is to just execute the “reboot” command.  VMs and all processes will be brought down gracefully as “reboot” waits for the underlying XenServer operating system to halt cleanly before power cycling automatically.

Finally, since the XenTools package has been updated, I make sure that after the stand-alone system powers backup that I apply the new XenTools to each VM.

 

 

Step 7.  How I apply a patch to a Pool

Let me state, again, that for a Pool a hotfix or service pack must be applied to all hosts.  If a host is updated in a Pool and the others are not, XAPI’s “state database” will eventually be shared amongst pool members and this is not good: it can cause a corrupt state database that is not repairable.

So, dedicate time to apply any hotfix or service pack to the Pool Master (first) and all other hosts within a Pool… in one shot.

 

XenCenter can be used to facilitate the application of a hotfix or service pack, but you have to have a licensed copy to do this.  From my own experience and desire for control, I prefer to install patches from the command line.  The choice is yours, but at least run through the checks discussed here for disk space.

 

From XenCenter:

I use the XenCenter -> Tools -> Updates to download and apply XS62ESP1 across my entire pool: letting the patch be applied to each host, accordingly, and rebooted (starting with the Pool Master first!)

 

From the Command Line:

The Pool Master must be updated first as it is the leader of the pack: no questions, no hesitations.

I download the hotfix or service pack after updating XenCenter (if required, such as it is to apply XenServer 6.2 SP1 successfully).  Decompressing the ZIP file, I can be assured that XSxxxxxx.update file is in-tact.  I then check my disk space on my root (sda) partition on my Pool Master:

 

cd /
df -h  <-- "disk filesystem, human readable"
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 4.0G 2.4G 1.4G 63%    /   <--- I have 1.4G available, so I am good albeit close!!
none 1.1G 4.0K 1.1G 1% /dev/shm /opt/xensource/packages/iso/XenCenter.iso

I clean out any “older” patch files from /var/patch and any “older” logs from /var/log on the Pool Master:

 

cd /var/log
rm -rf *gz
cd /var/patch
rm -rf *

I then secure copy the XS62ESP1.update file (using scp from a Linux host or WinSCP from a Windows host) to root’s home directory on the Pool Master:

 

scp XS62ESP1.update root@masterserver:/root/

 

Now I can start to apply the hotfix/service pack.  I can’t just jump in and install it as I need to make XAPI (and the Pool Master) aware of the hotfix/service pack so I can later deploy it to the Pool Members:

 

xe patch-upload filename=XS62ESP1.update
0850b186-4d47-11e3-a720-001b2151a503

 

It will spit out a UUID — 0850b186-4d47-11e3-a720-001b2151a503 –copy and paste this as you will need it to install the hotfix/service pack to all other Pool Members.  To find out the list of Pool Member UUIDs, starting with the Pool Master, I execute:

 

xe host-list
uuid ( RO) : 17aee7b6-9f36-4c97-8616-05799a5b303e <-- Pool Master UUID
 name-label ( RW): masterserver
 name-description ( RW): Default install of XenServer
uuid ( RO) : 23afe777-8cd2-83df-91b0-2235fcde2122 <-- Secondary Pool Member UUID, etc...
 name-label ( RW): secondaryserver
 name-description ( RW): Default install of XenServer

 

Now that XAPI is aware of this hotfix/service pack I can then install it on the Pool Master by using the “xe patch-apply” command along with the UUID of the hotfix/service pack that was displayed before and the UUID of my Pool Master:

 

xe patch-apply uuid=0850b186-4d47-11e3-a720-001b2151a503 host-uuid=17aee7b6-9f36-4c97-8616-05799a5b303e

 

This is a rather large patch, so wait and eventually you will see (do not reboot the Pool Master after installation is complete):

 

Preparing... ##################################################
 xen-device-model ##################################################
 Preparing... ##################################################
 xen-hypervisor ##################################################
 Preparing... ##################################################
 xen-tools ##################################################
 Preparing... ##################################################
 xen-firmware ##################################################
 Preparing... ##################################################
 blktap ##################################################
 Preparing... ##################################################
 sm ##################################################
 Preparing... ##################################################
 md3000-rdac-modules-xen-2.6.##################################################
 Preparing... ##################################################
 md3000-rdac-modules-kdump-2.##################################################
 Preparing... ##################################################
 kernel-xen ##################################################
 Preparing... ##################################################
 kernel-kdump ##################################################
 openvswitch-modules-xen-2.6.##################################################
 Preparing... ##################################################
 openvswitch-modules-kdump-2.##################################################
 Regenerating initrds for kernels
 Preparing... ##################################################
 xapi-core ##################################################
 Preparing... ##################################################
 xapi-xenopsd ##################################################
 Preparing... ##################################################
 xapi-networkd ##################################################
 Preparing... ##################################################
 elasticsyslog ##################################################
 Shutting down kernel logger: [ OK ]
 Shutting down system logger: [ OK ]
 Starting system logger: [ OK ]
 Starting kernel logger: [ OK ]
 Shutting down kernel logger: [ OK ]
 Shutting down system logger: [ OK ]
 Starting system logger: [ OK ]
 Starting kernel logger: [ OK ]
 Preparing... ##################################################
 hwdata ##################################################
 Preparing... ##################################################
 xapi-xe ##################################################
 Preparing... ##################################################
 perf-tools-rrdd-gpumon ##################################################
 Starting XCP RRDD plugin xcp-rrdd-gpumon: [ OK ]
 Preparing... ##################################################
 Stopping XCP RRDD plugin xcp-rrdd-iostat: [ OK ]
 Stopping XCP RRDD plugin xcp-rrdd-squeezed: [ OK ]
 Stopping XCP RRDD plugin xcp-rrdd-xenpm: [ OK ]
 perf-tools-rrdd-plugins ##################################################
 Starting XCP RRDD plugin xcp-rrdd-iostat: [ OK ]
 Starting XCP RRDD plugin xcp-rrdd-squeezed: [ OK ]
 Starting XCP RRDD plugin xcp-rrdd-xenpm: [ OK ]
 Preparing... ##################################################
 v6d ##################################################
 Preparing... ##################################################
 vgpu ##################################################
 Latest '/opt/xensource/packages/iso/xs-tools-6.2.0-3.iso'
 9bdefc9e-84b0-27e8-2a65-978bd6ff4e4e is the local tools SR: scanning
 Done

 

The Pool Master now has knowledge of the hotfix/service pack.  I then repeat my disk space checks on the next target host/Pool Member.

Once I am certain I have enough disk space and that old patches/logs are cleared from the Pool Member, I then execute the same “xe patch-apply” command, but this time specifying the Pool Member’s “host-uuid”:

 

xe patch-apply uuid=0850b186-4d47-11e3-a720-001b2151a503 host-uuid=17aee7b6-9f36-4c97-8616-05799a5b303e

 

This is a rather large patch and it is being transmitted over the network to the next Pool Member specified by the “host-uuid”, so be patient and eventually you will see (do not reboot the Pool Member or Pool Master):

 

Preparing... ##################################################
 xen-device-model ##################################################
 Preparing... ##################################################
 xen-hypervisor ##################################################
 Preparing... ##################################################
 xen-tools ##################################################
 Preparing... ##################################################
 xen-firmware ##################################################
 Preparing... ##################################################
 blktap ##################################################
 Preparing... ##################################################
 sm ##################################################
 Preparing... ##################################################
 md3000-rdac-modules-xen-2.6.##################################################
 Preparing... ##################################################
 md3000-rdac-modules-kdump-2.##################################################
 Preparing... ##################################################
 kernel-xen ##################################################
 Preparing... ##################################################
 kernel-kdump ##################################################
 openvswitch-modules-xen-2.6.##################################################
 Preparing... ##################################################
 openvswitch-modules-kdump-2.##################################################
 Regenerating initrds for kernels
 Preparing... ##################################################
 xapi-core ##################################################
 Preparing... ##################################################
 xapi-xenopsd ##################################################
 Preparing... ##################################################
 xapi-networkd ##################################################
 Preparing... ##################################################
 elasticsyslog ##################################################
 Shutting down kernel logger: [ OK ]
 Shutting down system logger: [ OK ]
 Starting system logger: [ OK ]
 Starting kernel logger: [ OK ]
 Shutting down kernel logger: [ OK ]
 Shutting down system logger: [ OK ]
 Starting system logger: [ OK ]
 Starting kernel logger: [ OK ]
 Preparing... ##################################################
 hwdata ##################################################
 Preparing... ##################################################
 xapi-xe ##################################################
 Preparing... ##################################################
 perf-tools-rrdd-gpumon ##################################################
 Starting XCP RRDD plugin xcp-rrdd-gpumon: [ OK ]
 Preparing... ##################################################
 Stopping XCP RRDD plugin xcp-rrdd-iostat: [ OK ]
 Stopping XCP RRDD plugin xcp-rrdd-squeezed: [ OK ]
 Stopping XCP RRDD plugin xcp-rrdd-xenpm: [ OK ]
 perf-tools-rrdd-plugins ##################################################
 Starting XCP RRDD plugin xcp-rrdd-iostat: [ OK ]
 Starting XCP RRDD plugin xcp-rrdd-squeezed: [ OK ]
 Starting XCP RRDD plugin xcp-rrdd-xenpm: [ OK ]
 Preparing... ##################################################
 v6d ##################################################
 Preparing... ##################################################
 vgpu ##################################################
 Latest '/opt/xensource/packages/iso/xs-tools-6.2.0-3.iso'
 9bdefc9e-84b0-27e8-2a65-978bd6ff4e4e is the local tools SR: scanning
 Done

 

 

Repeat this process for each Pool Member’s “host-uuid” and then reboot each host: starting with the Pool Master down to the last Pool Member.

 

 

APPENDIX OR “PATCH NOT INSTALLED”

So, let’s get to the circumstances that may cause the ambiguous “PATCH NOT INSTALLED” error (or similar error) message when applying XS62ESP1 and how to avoid/correct this.

 

Hello, Everyone and Happy New Year!

 

Speaking of “New Year”, I had read within our Citrix Discussion Forums where a few Citrix Community members reported difficulty with installing Service Pack 1 for XenServer 6.2.  I had such a great experience in my own tests that eventually, well: I wrote about it on December 31, 2013 – here (Blogs.Citrix.com) and also within the Citrix Discussion Forums.

Thanks to a suggestion by one of our community members, I have updated my original article: formatting, adding more details and using spell check (aren’t we using 1984′s Doublespeak already?), but most importantly, I am providing emphasis on a successful patch/service pack installation and how to work around some of the items a few community members have faced.

I’d like to call “Mathematically Improbable, but Possible Side Effects”.  The Community Member – you know who you are and I thank you – suggested  an “Appendix”.  Structurally, this person is right, but for the sake of redundancy I will link these two articles together and add the following as an Appendix to the original article.

 

So, let’s get to the circumstances that can cause the ambiguous “PATCH NOT INSTALLED” error (or similar error) message when applying XS62ESP1 and how to avoid/correct this.

 

0.  Eject all ISO images from VMs and ensure Syslog is set to “local” – not a remote collector

This has been added thanks to work with clients and information provided by the community.  In short, before applying a patch one wants all mounted ISO images to be ejected.  Likewise, Syslog preferences should be set to local (specific to the 6.2 SP1 patch).

To eject all ISO images that may be mounted, execute:

 

xe vm-cd-eject --multiple

 

There may be error messages reported, but these are GOOD because that means a VM had NO ISO MOUNTED.  Lastly, XenCenter/XenServer offers the ability to remote log all syslog messages.  Make sure – before applying a patch – that this option is disabled and set to “LOCAL”.  This can be done by selecting your XenServer in XenCenter and selecting “Properties”.  Under “Log Destination”, ensure this option is set to “LOCAL”.

 

 

1.  XenCenter is showing I have older XenServer patches to apply?

This is the most complex one to address, but I have only dealt with it twice and read about it once.

In short, this seems to be due to the fact that XenCenter was not updated before applying XenServer 6.2 SP1 to a stand-alone server or Pool.  The version of XenCenter that should be installed before applying Service Pack 1 to XenServer 6.2 is specifically version 6.2 Build 1377.

The solution is to shutdown XenCenter, download the latest XenCenter (for XenServer 6.2 SP1).  After doing so, you then need to gain access to the command line of your Pool Master or Stand-Alone host.  Run the following command to find out the UUIDs of, for example, the XenServer 6.1 patches that are being reported as “need to be installed” or “will be installed on reboot” (even though they won’t because they are not technically there – I am also using XS61 as an example for this could be XS602 or XS56, etc):

 

xe patch-list | grep XS61 -B 1

The output should be similar to:

 

--
uuid ( RO) : d9c753b9-a15b-4a31-897b-97fdae609031
name-label ( RO): XS61E009
--
uuid ( RO) : 8b0518d8-3610-4b7f-8a31-0e02718d2f9f
name-label ( RO): XS61E013

 

Now, from the Pool Master or Stand-Alone server, take each UUID that you see from the output and execute:

 

xe patch-destroy uuid=d9c753b9-a15b-4a31-897b-97fdae609031  <--- UUID of one of the "older/irrelevant" patches

 

Repeat the process until the “xe patch-list” command shows no more of the “older version” patches.  XenCenter should now quit worrying about irrelevant patches from older versions.

 

 

2.  “File System on Control Domain Full” / Disk Space

Checking each host for available disk space is key.  There is intelligence being added to patch installation to check disk space, however if installing from the command line or through XenCenter it is possible to run out of disk space.  The SP1 update – for me – went fine with 1.4Gig of free space.  However, I cleaned old patches out of /var/patch as well as any “older logs” from /var/log to ensure I had disk availability.

To recover from a failed patch install due to lack of disk space:

 

  1. Free disk space
  2. Clear logs
  3. Destroy the SP1 (or all patches associated with 6.2)

 

So, from the command-line, you can execute the following on the host in question since the Service Pack 1 patch contains a roll up of all previous patches::

 

cd /var/patch/
rm -rf *
cd /var/log/
rm -rf *gz

 

xe patch-list | grep uuid

 

xe patch-destroy uuid=<uuid for each UUID listed from the above command>

Once completed, re-apply the Service Pack (again, make sure that the latest version of XenCenter is installed first).

 

 

3.  Upgrading from a previous version of XenServer…

If upgrading from a previous version of XenServer to XenServer 6.2, do not apply 6.2 hotfixes.  Instead, download the latest XenCenter and apply Service Pack 1.

This is based on one rare instance where a handful of 6.2 patches were applied and then the application of Service Pack 1 failed.  This is most likely due to the disk space needed, but again – if upgrading to 6.2, upgrade to the Vanilla version and then apply Service Pack 1 after upgrading XenCenter.

 

 

And this is a note from My Virtual Desktop.

–jkbs