AWS re:Invent 2018: What Excited Us and What Surprised Us

With AWS re:Invent 2018 now behind us, we wanted to share some of our reactions to the event and many of the announcements. It was an exciting time for Fugue, as we announced the availability of our new SaaS solution, an easy-to-use solution for finding compliance violations in your AWS environments, detecting infrastructure drift, and automatically remediating it when it occurs.

Of course, there was no shortage of announcements from AWS, and engineers at Fugue were abuzz on Slack about many of the new services AWS introduced. So we wanted to share some of those with you here.

AWS Lambda Runtimes

Jasper Van der Jeugt, Principal Engineer
I am most excited about the custom AWS Lambda runtimes. I've previously used non-supported languages by including a binary linked for Lambda's platform, but this always felt a bit messy due to the need to serialize in between languages, among other things. I think the runtime API provides the capabilities to not only add new languages, but also have more granular resource control (e.g., when dealing with connection pools) for already supported languages.

Dan Kerrigan, Managing Senior Engineer
The Lambda serverless announcements stand out to me. While GCP Cloud Functions have just been launched and Azure Functions support just a handful of languages and service integrations, AWS already had rich Lambda service integration and just announced support for any language, including Rust and Ruby, and further service integration with Application Load Balancers. Those announcements reinforce an already strong serverless offering that include integration with API Gateway, Step Functions, CloudWatch events, S3, DynamoDB, and more. If serverless is what you want, there's no better cloud provider than AWS.

Curtis Myzie, Principal Software Engineer
Really exciting to see the new possibilities with Lambda. Moving forward it seems like this will provide another big boost to the rate of adoption of serverless architectures. With a growing serverless ecosystem I’ll be interested to see what other projects turn up in the coming year as a result. It definitely has helped us deliver our SaaS product quickly so we know it can be a boon for developers. Curious to see if the other cloud platforms follow suit. It’s priceless that Amazon has had Go support for a while now in Lambda while Google… the creator of Go… still does not have a GA version of Go Cloud Functions.

Building on the Lambda announcements are all the new Step Function integrations. If I was going to make a stateful application today in the cloud, there’s no question I would look to see if Step Functions could be used to simplify the implementation. It’s a great set of functionality you get out of the box with Step Functions and the possibilities just opened up even more. I personally want to try out triggering a state machine from an SQS message and running a task in Fargate.

For 2019 I’m looking forward to more community discussion on best practices surrounding serverless applications. It’s still a new area and everyone is still working to understand the limitations and how to best approach problems. Considering Fugue is heavily leveraging serverless, I look forward to contributing to the discussion.

Robert Schanafelt, Principal Software Engineer
The new Lambda custom runtimes will be awesome for deploying apps with whatever language or platform we choose to use (e.g., Haskell?). These improvements will not only be great for developers in general, I think these could prove useful for us at Fugue.

AWS Lambda Layers

James Mundia, Senior Customer Support Engineer
I love how strategic AWS is not only with their actual products and services, but how they take active steps to shape how people use the cloud. A great example of this is AWS Lambda Layers. I love Lambda functions because I can compartmentalize my application code to be extremely small and specific.

With Lambda Layers, I can now have a core business logic layer that the rest of the functions can use. This is a game changer for serverless development, as you can use DRY concepts to speed up development. Microservice development will start to look a lot more like “traditional” dev workflows, where I can simply import dependencies that I need, when I need them.

Firecracker - Lightweight Virtualization for Serverless Computing

Owen Diehl, Software Engineer
Firecracker: It’s very exciting to see languages relying on ML-style type systems. This brings a bunch of compile-time safety through provable guarantees about code. It’s particularly exciting in Rust’s case because this yields benefits without having to buy into fully functional code. At Fugue, we’re obviously strong proponents of type safety (see Ludwig). Firecracker being built in Rust is important because it’s another large scale demonstration of value of the language’s design choices. It serves as a good reminder that the quality of security- and performance-oriented systems programming can be elevated fairly easily via type safety.

Mike Lin, Senior Software Engineer
The announcement coming out of AWS re:Invent that excited me the most was Firecracker, for three reasons. First, it's an exciting development in the state of the art of virtualization. Second, and more interestingly, it's a glimpse at how AWS Lambda runs under the hood. Third, and most interestingly, it's a new open-source project shepherded by AWS. It'll be fascinating to see how the project develops and how AWS continues to interact with the open-source community in the future.

DynamoDB Announcements

Robert Schanafelt
DynamoDB Transactions and On-Demand as well as Lambda Layers and ALB invocation are what excited me the most. The DynamoDB features will help reduce costs and make the development of distributed systems even better. Lambda Layers will make the development of complex applications more streamlined by being able to share and reuse "layers" of functionality. ALB invocation of Lambdas should simplify some use cases for deploying functions.

James Mundia
DynamoDB Transactions is another great example of AWS listening to customer use cases and developing solutions. DDB is great for building performant NoSQL databases. And while NoSQL is gaining in popularity for large use cases, there are some concerns for specific types of workflows, like financial transactions, which need to be atomic.

DDB Transactions allow for the development of data workflows that require specific levels of atomicity. By adding two new DDB operation types, you now have access to batch operations to ensure that database operations meet the requirements for things like coordinated inserts, deletes, and updates.

This is a big move by AWS, essentially opening up DDB to more enterprise-level workflows. With the promise of ACID databases, developers can feel much more comfortable building large-scale, production-ready applications while taking advantage of DDB's cost savings and performance.

Tyler Drombosky, Managing Principal Engineer
More than the past few years, AWS really went back to its roots in delivering value to its customers at re:Invent 2018. I’ve always found using AWS to be its best when they provide robust building blocks for deploying applications. These aren’t always the most glamorous features, but they’re the technologies you’re happy you have when you’re in the weeds trying to get something out the door.

I’ve personally battled with scaling DynamoDB access for new applications and wrangled ACID-like capabilities on top of multi-key updates. AWS’s new announcements remove those challenges. Our new product relies heavily on Lambda. We’re already taking advantage of CloudWatch Log Insights and I can’t wait to see if Layers can simplify our deployments.

What always made AWS appealing to me is the ability to build whatever I want however I want and only needing to focus on the technical minutia when needed. This new round of deeply technical announcements makes me happy that AWS is still focused on providing a great developer experience.

CloudWatch Logs Insights

James Mundia
Prior to the CloudWatch Logs Insights announcement, I had built up a lot of technical debt trying to build a log management solution for the Fugue Support team. This announcement from AWS solves a huge business need, the ability to easily get log insight from your applications. And CloudWatch Logs Insights is fully managed, so I don’t have to set anything up! Just pick a log group in CloudWatch, write queries (or use the samples provided), and I’ve got log analytics and visualization.

A Greater Focus on AWS Security

Kate Green, Senior Software Engineer
AWS re:Invent has me thinking a lot about efforts around providing better security. Security is hard, but the industry is fast coming up with ways to automate the things we should all be doing in our applications and infrastructure. This is a welcome development, as attacks happen most often by exploiting misconfiguration. When a system gets hacked, it's typically either a bad configuration or an unpatched system containing an obvious flaw. Seeing solutions that automate catching these sorts of errors is heartening for everyone's data security.

VMWare, AWS Outposts, and the Sheer Size of AWS

Drew Wright, Co-Founder & VP Communications
What always strikes me with every AWS re:Invent is just how much bigger AWS keeps getting, and the event serves as a decent proxy for this. That said, Lydia Leong, VP and Distinguished Analyst with Gartner, makes this point better than most: 

Reflection from #reinvent: This year I have *personally* reviewed more dollars in combined first-year contract value in client AWS enterprise discount agreements than the total expected 2018 revenue of Azure+GCP+Oracle Cloud Infrastructure+IBM Cloud (minus SaaS) put together.

We all know that AWS is the leading cloud provider, but Lydia’s tweet is incredible. And considering the emphasis on AWS’s partnership with VMware and their announcement of AWS Outposts, AWS is serious about not only bringing the enterprise to the cloud, but bringing AWS to the enterprise datacenter.

AWS Ground Station was the Big Surprise

James Mundia
The AWS Ground Station announcement doesn’t have any professional bearing on me, but I think it’s just really cool. This service allows you to "Ingest and Process Data from Orbiting Satellites." My favorite bit from the announcement is this snippet:

"Your entire workflow, from the ground stations all the way through to processing, storage, reporting, and delivery, can now be done on elastic, pay-as-you-go infrastructure!"

AWS is taking its cloud pricing and workflow model literally to the clouds, and beyond!

Robert Schanafelt
AWS Ground Station was the most surprising. On-demand satellite data downlinks? I didn't see that one coming.

_________________________

The dust has barely settled, but we're already looking forward to AWS re:Invent 2019 and the inevitable announcements between now and then.