Call us @ +30 2541068185

Thinking the cloud is magical with 100% uptime ? Think again.

Amazon will tomorrow begin a bloody global reboot of its Elastic Compute Cloud (EC2) compute instances after it found a security bug within the Xen virtualisation platform.

The rolling minutes-long reboots would be completed by 30 September. Amazon did not name the reason for the upgrade, widely thought to be a security issue affecting underlying hosts.

Tech site ITNews cited an unnamed source who said the reboot was due to an unspecified vulnerability in the open-source Xen-108 platform. Later, Xen and Amazon confirmed a fix for a non-disclosed security flaw is due to be released on October 1.

Reboots made prior to the patch blitz would not guarantee connection to a patched host unlike previous maintenance updates.

Thorsten von Eicken, founder of Rightscale which manages AWS work loads, said EC2 users should monitor their ‘events’ page within the AWS console for the most reliable updates.

“For instances where a short reboot is safe and acceptable, you don’t need to do anything: They will simply reboot during maintenance and stay on the same host with the same ephemeral disks and the same IP address,” von Eicken said.

“For databases, if you have set up the recommended master-slave configuration across AZs, you have the option to reboot the impacted AZ ahead of the maintenance window in an attempt to get an instance that is already patched.”

Back in 2011  Randy Bias has blogged about Amazon mandating instance reboots for hundreds, perhaps thousands, of instances (Amazon’s term for VMs). Affected instances seem to be scheduled for reboots over the next couple of weeks. Speculation is that the reboots are to patch a recently-reported vulnerability in the Xen hypervisor, which is the virtualization technology that underlies Amazon’s EC2. The GigaOm story gives some links, and the CRN story discusses customer pain.

Maintenance reboots are not new on Amazon, and are detailed on Amazon’s documentation about scheduled maintenance. The required reboots this time are instance reboots, which are easily accomplished — just point-and-click to reboot on your own schedule rather than Amazon’s (although you cannot delay past the scheduled reboot). Importantly, instance reboots do not result in a change of IP address nor do they erase the data in instance storage (which is normally non-persistent).

For some customers, of course, a reboot represents a headache, and it results in several minutes of downtime for that instance. Also, since this is peak retail season, it is already a sensitive, heavy-traffic time for many businesses, so the timing of this widespread maintenance is problematic for many customers.

However, cloud IaaS isn’t magical. If these customers were using dedicated hosting, they would still be subject to mandated reboots for security patches — hosting providers generally offer some flexibility on scheduling such reboots, but not aa lot (and sometimes none at all if there’s an exploit in the wild). If these customers were using a provider that uses live migration technology (like VMotion on a VMware-virtualized cloud), they might be spared reboots for system reasons, but they might still be subject to reboots for mandated operating system patches.

Given that what’s underlying EC2 are ordinary physical servers running virtualization without a live migration technology in use, customers should reasonably expect that they will be subject to reboots — server-level (what Amazon calls a system reboot), as well as instance-level — and also anticipate that they may sometimes need to reboot for their own guest OS patches and the like (assuming that they don’t simply patch their AMIs and re-launch their instances, arguably a more “cloudy” way to approach this problem).

What makes this rolling scheduled maintenance remarkable is its sheer scale. Hosting providers typically have a few hundred customers and a few thousand servers. Mass-market VPS hosters have lots of VPS containers, but there’s a roughly 1:1 VPS:customer ratio and a small-business-centricity that doesn’t lead to this kind of hullabaloo. Amazon’s largest competitor is estimated to be around the 100,000 VM mark. Only the largest cloud IaaS providers have more than 2,000 VMs. Consequently, this involves a virtually unprecedented number of customers and mission-critical systems.

Amazon has actually been very good about not taking down its cloud customers for extended maintenance windows. (I can think of one major Amazon competitor that took down one whole data center for an eight-hour maintenence evidently involving a total outage this past weekend, and which regularly has long-downtime maintenance windows in general.) A reboot is an inconvenience, but if you are running production infrastructure, you should darn well think about how to handle the occasional reboot, including reboots that affect a significant percentage of your infrastructure, because reboots are not likely to go away in IaaS anytime soon.

To hammer on the point again: Cloud IaaS is not magical. It still requires management, and it still has some of the foibles of both physical servers and non-cloud virtualization. Being able to push a button and get infrastructure is nice, but the responsibility to manage that infrastructure doesn’t go away — it’s just that many cloud customers manage to delay the day of reckoning when the attention they haven’t paid to management comes back to bite them.

If you run infrastructure, regardless of whether it’s in your own data center, in hosting, or in cloud IaaS, you should have a plan for “what happens if I need to mass-reboot my servers?” because it is something that will happen. And add “what if I have to do that immediately?” to the list, because that is also something that will happen, because mass exploits and worms certainly have not gone away.

153 thoughts on “The Cloud is not magical”

  1. Pingback: 8778
  2. Pingback: writeessay
  3. Pingback: Generic cialis
  4. Pingback: Viagra generika
  5. Pingback: Cialis generique
  6. Pingback: Generic viagra
  7. Pingback: Cialis online
  8. Pingback: continue
  9. Pingback: Generic cialis
  10. Pingback: Generic cialis
  11. Pingback: page
  12. Pingback: read further
  13. Pingback: read completely
  14. Pingback: click
  15. Pingback: more info
  16. Pingback: click to continue
  17. Pingback: follow this post
  18. Pingback: Cialis generic
  19. Pingback: Buy cialis
  20. Pingback: Cialis 5 mg
  21. Pingback: Generic cialis
  22. Pingback: learn more here
  23. Pingback: follow this post
  24. Pingback: notehub.org
  25. Pingback: go to the source
  26. Pingback: notehub.org
  27. Pingback: notehub.org go Wlq
  28. Pingback: notehub.org
  29. Pingback: notehub.org
  30. Pingback: notehub.org
  31. Pingback: more on the page
  32. Pingback: notehub.org
  33. Pingback: notehub.org
  34. Pingback: see more
  35. Pingback: read
  36. Pingback: find out more here
  37. Pingback: d2a20854899.com
  38. Pingback: topfuckgals.mobi
  39. Pingback: topdatingse.mobi
  40. Pingback: freedpgals.mobi
  41. Pingback: read further
  42. Pingback: solars.biz
  43. Pingback: sildenafil 100mg
  44. Pingback: simvastatin 20 mg
  45. Pingback: sildenafil citrate
  46. Pingback: biaxin generic
  47. Pingback: biaxin antibiotic
  48. Pingback: biaxin coupon
  49. Pingback: biaxin coupons
  50. Pingback: fluoxetine hcl
  51. Pingback: fluoxetine 20 mg
  52. Pingback: fluoxetine 10 mg
  53. Pingback: alprostadil
  54. Pingback: alprostadil price
  55. Pingback: buy cialis
  56. Pingback: 2019
  57. Pingback: nsfwtophookup.eu
  58. Pingback: NSFWTOPHOOKUP
  59. Pingback: cleantalkorg2.ru
  60. Pingback: #macron #Lassalle
  61. Pingback: a2019-2020
  62. Pingback: facebook
  63. Pingback: facebook1
  64. Pingback: javsearch.mobi
  65. Pingback: cialis
  66. Pingback: generic cialis
  67. Pingback: sildenafil
  68. Pingback: tadalafil
  69. Pingback: cheap cialis
  70. Pingback: sildenafil 20 mg
  71. Pingback: sildenafil 100
  72. Pingback: buy biaxin
  73. Pingback: buy ceftin
  74. Pingback: buy chloromycetin
  75. Pingback: buy biaxin online
  76. Pingback: buy ceftin online
  77. Pingback: chloromycetin
  78. Pingback: buy cordarone
  79. Pingback: sildenafil tablets
  80. Pingback: sildenafil generic
  81. Pingback: sildenafil 50 mg
  82. Pingback: sildenafil coupons
  83. Pingback: sildenafil 100 mg
  84. Pingback: bactrim ds
  85. Pingback: ceftin antibiotic
  86. Pingback: cephalexin 500mg
  87. Pingback: fluconazole 150 mg
  88. Pingback: cefdinir 300 mg
  89. Pingback: bactrim antibiotic
  90. Pingback: tadalafila
  91. Pingback: watchnudefree.eu
  92. Pingback: cymbalta generic
  93. Pingback: cymbalta reviews
  94. Pingback: vardenafil
  95. Pingback: duloxetine
  96. Pingback: vardenafil 20 mg
  97. Pingback: duloxetine 20 mg
  98. Pingback: vardenafil 20mg
  99. Pingback: duloxetine hcl
  100. Pingback: duloxetine 60 mg

Leave a Reply