Handling AWS Spot Instance Termination Notices

On the very worth-your-time-to-follow official Amazon Web Services blog, Jeff Barr announced today that EC2 spot instances will now get "two-minute warning" termination notices.

 

The long and short of it is that you can now query instance metadata or the DescribeSpotInstanceRequests API to find out if a spot instance has been marked for termination. At that point, you'll have two minutes to react and shut down cleanly. This gives your application time to wrap up any work in progress.

 

In the post, Jeff describes the mechanism in more detail, but it is basically an instance metadata endpoint that returns a 404 HTTP status code until the instance is marked for termination. AWS recommends polling this endpoint every five seconds. This can be done in a variety of ways, but one of the easiest to start experimenting with is to periodically curl the endpoint and trigger shell scripts upon notification.

 

An obvious first resort for polling something and running a shell script on change is crond, but cron only provides minute increments for job scheduling. So, a simple alternative is to just background a shell script (using rc.local, supervisord, daemontools, or whatever you like) that polls every five seconds and runs your shutdown hook script when it detects the notification.

 

<code class="language-bash"><span class="hljs-shebang">#!/bin/bash</span><span class="hljs-keyword">while</span> <span class="hljs-literal">true</span>    <span class="hljs-keyword">do</span>        <span class="hljs-keyword">if</span> [ -z $(curl -Is http://<span class="hljs-number">169.254</span>.<span class="hljs-number">169.254</span>/latest/meta-data/spot/termination-time | head -<span class="hljs-number">1</span> | grep <span class="hljs-number">404</span> | cut <span class="hljs-operator">-d</span>   <span class="hljs-operator">-f</span> <span class="hljs-number">2</span>) ]            <span class="hljs-keyword">then</span>                logger <span class="hljs-string">"Running shutdown hook."</span>                <span class="hljs-comment"># Call your shutdown script here.</span>                <span class="hljs-built_in">break</span>            <span class="hljs-keyword">else</span>                <span class="hljs-comment"># Spot instance not yet marked for termination.</span>                sleep <span class="hljs-number">5</span>        <span class="hljs-keyword">fi</span>    <span class="hljs-keyword">done</span></code>

 

This is a simple implementation, but it may be just what you need if you have a simple application shutdown hook. A more robust tool, Shudder, helps with graceful shutdowns in autoscaling groups; that is expected to add support for spot termination notifications soon.

We're excited about spot termination notices, and we will definitely be using them in our Fugue product.

 

Categorized Under

AWS

Secure Your Cloud

Find security and compliance violations in your cloud infrastructure and ensure they never happen again.