About Us

We're Working On Some Of Cloud Computing's Toughest Challenges

Fugue automates cloud operations, eases public and private sector compliance, and simplifies lifecycle management of the AWS infrastructure service stack. To ensure customers can innovate faster, Fugue validates systems before they're built and continuously enforces them after. Fugue has eight patents granted and 16 pending. Gartner named Fugue a Cool Vendor in Cloud Computing 2017.

Our founders started Fugue in 2013 to face one of cloud computing’s most vexing challenges—the nitty-gritty, secure management of cloud infrastructure operations for businesses large and small. Since then, we’ve built a new kind of system for operating the cloud that automates infrastructure-as-code and policy-as-code from one source of truth. Our devoted team of more than 70 skilled engineers and talented creative professionals combines decades of experience with fresh vision and relentless quality control to deliver an innovative approach that works, while adhering to our shared cultural principles.

Our offices are primarily based in Frederick, MD, Washington, DC, and Silicon Valley, CA. Our team spans the globe, with members located in Atlanta, Miami, and Seattle and as far away as Zurich and Tokyo, all collaborating daily. We’re venture-backed, having raised $74MM to date, with prescient, supportive partners at New Enterprise Associates, Future Fund, The Maryland Venture Fund, and Core Capital. Meet our executive team below!

Our Leadership

Phillip Merrick
Chief Executive Officer
Phillip Merrick
Josh Stella
Co-founder & Chief Technology Officer
Josh Stella
Mary Alexander
Vice President of Sales
Mary Alexander
Gus Bessalel
Chief Financial Officer
Gus Bessalel
Nathan McCourtney
Vice President of Engineering
Nathan McCourtney
Richard Park
Vice President of Product
Richard Park
Tim Webb
Chief Strategy & Security Officer
Tim Webb
Heather Wiley
Chief People Officer
Heather Wiley
Andrew Wright
Co-founder & Vice President of Communications
Andrew Wright

Our Advisors

Frank Slootman
ServiceNow
Ben Fathi
Cloudflare & VMware
Dave Merkel
Expel & FireEye
Chad Fowler
Microsoft & Wunderlist
Joe Payne
Code42 & Eloqua
Amena Ali
VividCortex & Earth Networks

Our Investors

OUR OFFICES

Where Are We Located?

  • Location Details

    Fugue Offices

    San Jose, CA

  • Location Details

    Fugue Offices

    Washington, DC

  • Location Details

    Fugue Offices

    Miami, FL

  • Location Details

    Fugue Offices

    Atlanta, GA

  • Location Details

    Fugue Offices

    Baldwin City, KS

  • Location Details

    Fugue Headquarters

    Frederick, MD

  • Location Details

    Fugue Offices

    Manalapan, NJ

  • Location Details

    Fugue Offices

    Eugene, OR

  • Location Details

    Fugue Offices

    Seattle, WA

Career Opportunities

Join The Fugue Team

From Our Blog

Featured Articles

  • A CEO's Guide to Emacs

    Years—no, decades—ago, I lived in Emacs. I wrote code and documents, managed email and calendar, and shelled all in the editor/OS. I was quite happy. Years went by and I moved to newer, shinier things. As a result, I forgot how to do tasks as basic as efficiently navigating files without a mouse. About three months ago, noticing just how much of my time was spent switching between applications and computers, I decided to give Emacs another try. It was a good decision for several reasons that will be covered in this post. Covered too are .emacs and Dropbox tips so that you can set up a good, movable environment.   For those who haven't used Emacs, it's something you'll likely hate, but may love. It's sort of a Rube Goldberg machine the size of a house that, at first glance, performs all the functions of a toaster. That hardly sounds like an endorsement, but the key phrase is "at first glance." Once you grok Emacs, you realize that it's a thermonuclear toaster that can also serve as the engine for... well, just about anything you want to do with text. When you think about how much your computing life revolves around text, this is a rather bold statement. Bold, but true.   Perhaps more importantly to me though, it's the one application I've ever used that makes me feel like I really own it instead of casting me as an anonymous "user" whose wallet is cleverly targeted by product marketing departments in fancy offices somewhere near Soma or Redmond. Modern productivity and authoring applications (e.g., Pages or IDEs) are like carbon fiber racing bikes. They come kitted out very nicely and fully assembled. Emacs is like a box of classic Campagnolo parts and a beautiful lugged steel frame that's missing one crank arm and a brake lever that you have to find in some tiny subculture on the Internet. The first one is faster and complete. The second is a source of endless joy or annoyance depending on your personality—and will last until your dying day. I'm the sort of person who feels equal joy at finding an old stash of Campy parts or tweaking my editor with eLisp. YMMV.   A 1933 steel bicycle that I still ride. Check out this comparison of frame tubes: https://www.youtube.com/watch?v=khJQgRLKMU0.   This may give the impression that Emacs is anachronistic or old-fashioned. It's not. It's powerful and timeless, but demands that you patiently understand it on its terms. The terms are pretty far off the beaten path and seem odd, but there is a logic to them that is both compelling and charming. To me, Emacs feels like the future rather than the past. Just as the lugged steel frame will be useful and comfortable in decades to come and the carbon fiber wunderbike will be in a landfill, having shattered on impact, so will Emacs persist as a useful tool when the latest trendy app is long forgotten.   If the notion of building your own personal working environment by editing Lisp code and having that fits-like-a-glove environment follow you to any computer is appealing to you, you may really like Emacs. If you like the new and shiny and want to get straight to work without much investment of time and mental cycles, it's likely not for you. I don't write code any more (other than Ludwig and Emacs Lisp), but many of the engineers at Fugue use Emacs to good effect. I'd say our engineers are about 30% Emacs, 40% IDEs, and 30% Vim users. But, this post is about Emacs for CEOs and other Pointy-Haired Bosses (PHB)1 (and, hey, anyone who’s curious), so I'm going to explain and/or rationalize why I love it and how I use it. I also hope to provide you with enough detail that you can have a successful experience with it, without hours of Googling.   Lasting Advantages   The long-term advantages that come with using Emacs just make life easier. The net gain makes the initial lift entirely worthwhile. Consider these:   No More Context Switching   Org Mode alone is worth investing some serious time in, but if you are like me, you are usually working on a dozen or so documents—from blog posts to lists of what you need to do for a conference to employee reviews. In the modern world of computing, this generally means using several applications, all of which have distracting user interfaces and different ways to store, sort, and search. The result is that you need to constantly switch mental contexts and remember minutiae. I hate context switching because it is an imposition put on me due to a broken interface model2 and I hate having to remember things my computer should remember for me in any rational world. In providing a single environment, Emacs is even more powerful for the PHB than the programmer, since programmers tend to spend a greater percentage of their day in a single application. Switching mental contexts has a higher cost than is often apparent. OS and application vendors have tarted up interfaces to distract us from this reality. If you’re technical, having access to a powerful language interpreter in a single keyboard shortcut (M-:) is especially useful.3   Many applications can be full screened all day and used to edit text. Emacs is singular because it is both an editor and a Lisp interpreter. In essence, you have a Turing complete machine a keystroke or two away at all times, while you go about your business. If you know a little or a lot about programming, you'll recognize that this means you can do anything in Emacs. The full power of your computer is available to you in near real time while you work, once you have the commands in memory. You won't want to re-create Excel in eLisp, but most things you might do in Excel are smaller in scope and easy to accomplish in a line or two of code. If I need to crunch numbers, I'm more likely to jump over to the scratch buffer and write a little code than open a spreadsheet. Even if I have an email to write that isn't a one-liner, I'll usually just write it in Emacs and paste it into my email client. Why context switch when you can just flow? You might start with a simple calculation or two, but, over time, anything you need computed can be added with relative ease to Emacs. This is perhaps unique in applications that also provide rich features for creating things for other humans. Remember those magical terminals in Isaac Asimov's books? Emacs is the closest thing I've encountered to them.4 I no longer decide what app to use for this or that thing. Instead, I just work. There is real power and efficiency to having a great tool and committing to it.   Creating Things in Peace and Quiet   What’s the end result of having the best text editing features I've ever found? Having a community of people making all manner of useful additions? Having the full power of Lisp a keychord away? It’s that I use Emacs for all my creative work, aside from making music or images.   I have a dual monitor set up at my desk. One of them is in portrait mode with Emacs full screened all day long. The other one has web browsers for researching and reading; it usually has a terminal open as well. I keep my calendar, email, etc., on another desktop in OS X, which is hidden while I'm in Emacs, and I keep all notifications turned off. This allows me to actually concentrate on what I'm doing. I've found eliminating distractions to be almost impossible in the more modern UI applications due to their efforts to be helpful and easy to use. I don't need to be constantly reminded how to do operations I've done tens of thousands of times, but I do need a nice, clean white sheet of paper to be thoughtful. Maybe I'm just bad at living in noisy environments due to age and abuse, but I’d suggest it’s worth a try for anyone. See what it's like to have some actual peace and quiet in your computing environment. Of course, lots of apps now have modes that hide the interface and, thankfully, both Apple and Microsoft now have meaningful full-screen modes. But, no other application is powerful enough to “live in” for most things. Unless you are writing code all day or perhaps working on a very long document like a book, you're still going to face the noise of other apps. Also, most modern applications seem simultaneously patronizing and lacking in functionality and usability.5 The only applications I dislike more than office apps are the online versions.   My desktop arrangement. Emacs on the left.   But what about communicating? The difference between creating and communicating is substantial. I'm much more productive at both when I set aside distinct time for each. We use Slack at Fugue, which is both wonderful and hellish. I keep it on a messaging desktop alongside my calendar and email, so that, while I'm actually making things, I'm blissfully unaware of all the chatter in the world. It takes just one Slackstorm or an email from a VC or Board Director to immediately throw me out of my work. But, most things can usually wait an hour or two.   Taking Everything with You and Keeping It Forever   The third reason I find Emacs more advantageous than other environments is that it's easy to take all your stuff with you. By this, I mean that, rather than having a plethora of apps interacting and syncing in their own ways, all you need is one or two directories syncing via Dropbox or the like. Then, you can have all your work follow you anywhere in the environment you have crafted to suit your purposes. I do this across OS X, Windows, and sometimes Linux. It's dead simple and reliable. I've found this capability to be so useful that I dread dealing with Pages, GDocs, Office, or other kinds of files and applications that force me back into finding stuff somewhere on the filesystem or in the cloud.   The limiting factor in keeping things forever on a computer is file format. Assuming that humans have now solved the problem of storage 6 for good, the issue we face over time is whether we can continue to access the information we've created. Text files are the most long-lived format for computing. You easily can open a text file from 1970 in Emacs. That’s not so true for Office applications. Text files are also nice and small—radically smaller than Office application data files. As a digital pack rat and as someone who makes lots of little notes as things pop into my head, having a simple, light, permanent collection of stuff that is always available is important to me.   If you’re feeling ready to give Emacs a try, read on! The sections that follow don’t take the place of a full tutorial, but will have you operational by the time you finish reading.   Learning To Ride Emacs - A Technical Setup   The price of all this power and mental peace and quiet is that you have a steep learning curve with Emacs and it does everything differently than you're used to. At first, this will make you feel like you’re wasting time on an archaic and strange application that the modern world passed by. It’s a bit like learning to ride a bicycle7 if you've only driven cars.   Which Emacs?   I use the plain vanilla Emacs from GNU for OS X and Windows. You can get the OS X version at http://emacsformacosx.com/ and the Windows version at http://www.gnu.org/software/emacs/. There are a bunch of other versions out there, especially for the Mac, but I've found the learning curve for doing powerful stuff (which involves Lisp and lots of modes) to be much lower with the real deal. So download it, and we can get started!8   First, You'll Need To Learn How To Navigate   I use the Emacs conventions for keys and combinations in this document. These are 'C' for control, 'M' for meta (which is usually mapped to Alt or Option), and the hyphen for holding down the keys in combination. So C-h t means to hold down control and type h, then release control and type t. This is the command for bringing up the tutorial, which you should go ahead and do.   Don't use the arrow keys or the mouse. They work, but you should give yourself a week of using the native navigation commands in Emacs. Once you have them committed to muscle memory, you'll likely enjoy them and miss them badly everywhere else you go. The Emacs tutorial does a pretty good job of walking you through them, but I'll summarize so you don't need to read the whole thing. The boring stuff is that, instead of arrows, you use C-b for back, C-f for forward, C-p for previous (up), and C-n for next (down). You may be thinking "why in the world would I do that, when I have perfectly good arrow keys?" There are several reasons. First, you don't have to move your hands from the typing position, and the forward and back keys used with Alt (or Meta in Emacspeak) navigate a word at a time. This is more handy than is obvious. The third good reason is that, if you want to repeat a command, you can precede it with a number. I often use this when editing documents by estimating how many words I need to go back or lines up or down and doing something like C-9 C-p or M-5 M-b. The other really important navigation commands are based on a for the beginning of a thing and e for the end of a thing. Using C-a|e are on lines, and using M-a|e, are on sentences. For the sentence commands to work properly, you'll need to double space after periods, which simultaneously provides a useful feature and takes a shibboleth of opinion off the mental table. If you need to export the document to a single space publication environment, you can write a macro in moments to do so.   It genuinely is worth going through the tutorial that ships with Emacs. I'll cover a few important commands for the truly impatient, but the tutorial is gold. Reminder: C-h t for the tutorial.   Learn To Copy and Paste   You can put Emacs into CUA mode, which will work in familiar ways, but the native Emacs way is pretty great and plenty easy once you learn it. You mark regions (like selecting) by using Shift with the navigation commands. So C-F selects one character forward from the cursor, etc. You copy with M-w, you cut with C-w, and you paste with C-y. These are actually called killing and yanking, but it's very similar to cut and paste. There is magic under the hood here in the kill ring, but for now, just worry about cut, copy, and paste. If you start fumbling around at this point, C-x u is undo...   Next, Learn Ido Mode   Trust me. Ido makes working with files much easier. You don't generally use a separate Finder|Explorer window to work with files in Emacs. Instead you use the editor's commands to create, open, and save files. This is a bit of a pain without Ido, so I recommend installing it before learning the other way. Ido comes with Emacs beginning with version 22, but you'll want to make some tweaks to your .emacs file so that it is always used. This is a good excuse to get your environment set up.   Most features in Emacs come in modes. To install any given mode, you'll need to do two things. Well, at first you'll need to do a few extra things, but these only need to be done once, and thereafter only two things. So the extra things are that you'll need a single place to put all your eLisp files and you'll need to tell Emacs where that place is. I suggest you make a single directory in, say, Dropbox that is your Emacs home. Inside this, you'll want to create an .emacs file and an .emacs.d directory. Inside the .emacs.d, make a directory called lisp. So you should have:   home|+.emacs|-.emacs.d | -lisp   You'll put the .el files for things like modes into the home/.emacs.d/lisp directory, and you'll point to that in your .emacs like so:   (add-to-list 'load-path "~/.emacs.d/lisp/")   Ido Mode comes with Emacs, so you won't need to put an .el file into your Lisp directory for this, but you'll be adding other stuff soon that will go in there.   Symlinks are Your Friend   But wait, that says that .emacs and .emacs.d are in your home directory, and we put them in some dumb folder in Dropbox! Correct. This is how you make it easy to have your environment anywhere you go. Keep everything in Dropbox and make symbolic links to .emacs, .emacs.d, and your main document directories in ~. On OS X, this is super easy with the ln -s command, but on Windows this is a pain. Fortunately, Emacs provides an easy alternative to symlinking on Windows, the HOME environment variable. Go into Environment Variables in Windows (as of Windows 10, you can just hit the Windows key and type "Environment Variables" to find this with search, which is the best part of Windows 10), and make a HOME environment variable in your account that points to the Dropbox folder you made for Emacs. If you want to make it easy to navigate to local files that aren't in Dropbox, you may instead want to make a symbolic link to the Dropbox Emacs home in your actual home directory.   So now you've done all the jiggery-pokery needed to get any machine pointed to your Emacs setup and files. If you get a new computer or use someone else's for an hour or a day, you get your entire work environment. This seems a little difficult the first time you do it, but it's about a ten minute (at most) operation once you know what you're doing.   But we were configuring Ido...   C-x C-f and type ~/.emacs RET RET to create your .emacs file. Add these lines to it:   ;; set up ido mode(require `ido)(setq ido-enable-flex-matching t)(setq ido-everywhere t)(ido-mode 1)   With the .emacs buffer open, do an M-x evaluate-buffer command, and you'll either get an error if you munged something or you'll get Ido. Ido changes how the minibuffer works when doing file operations. There is great documentation on it, but I'll point out a few tips. Use the ~/ effectively; you can just type ~/ at any point in the minibuffer and it'll jump back to home. Implicit in this is that you should have most of your stuff a short hop off your home. I use ~/org for all my non-code stuff and ~/code for code. Once you’re in the right directory, you'll often have a collection of files with different extensions, especially if you use Org Mode and publish from it. You can type period and the extension you want no matter where you are in the file name and Ido will limit the choices to files with that extension. For example, I'm writing this blog post in Org Mode, so the main file is:   ~/org/blog/emacs.org   I also occasionally push it out to HTML using Org Mode publishing, so I've got an emacs.html file in the same directory. When I want to open the Org file, I will type:   C-x C-f ~/o[RET]/bl[RET].or[RET]   The [RET]s are me hitting return for auto completion for Ido Mode. So, that’s 12 characters typed and, if you're used to it, a lot less time than opening Finder|Explorer and clicking around. Ido Mode is plenty useful, but really is a utility mode for operating Emacs. Let's explore some modes that are useful for getting work done.   Fonts and Styles   I recommend getting the excellent input family of typefaces for use in Emacs. They are customizable with different braces, zeroes, and other characters. You can build in extra line spacing into the font files themselves. I recommend a 1.5X line spacing and using their excellent proportional fonts for code and data. I use Input Serif for my writing, which has a funky but modern feel. You can find them on http://input.fontbureau.com/ where you can customize to your preferences. You can manually set the fonts using menus in Emacs, but this puts code into your .emacs file and, if you use multiple devices, you may find you want some different settings. I've set up my .emacs to look for the machine I'm using by name and configure the screen appropriately. The code for this is:   ;; set up fonts for different OSes. OSX toggles to full screen.(setq myfont "InputSerif")(cond ((string-equal system-name "Sampo.local") (set-face-attribute 'default nil :font myfont :height 144) (toggle-frame-fullscreen)) ((string-equal system-name "Morpheus.local") (set-face-attribute 'default nil :font myfont :height 144)) ((string-equal system-name "ILMARINEN") (set-face-attribute 'default nil :font myfont :height 106)) ((string-equal system-name "UKKO") (set-face-attribute 'default nil :font myfont :height 104)))   You should replace the system-name values with what you get when you evaluate (system-name) in your copy of Emacs. Note that on Sampo (my MacBook), I also set Emacs to full screen. I'd like to do this on Windows as well, but Windows and Emacs don't really love each other and it always ends up in some wonky state when I try this. Instead, I just fullscreen it manually after launch.   I also recommend getting rid of the awful toolbar that Emacs got sometime in the 90s when the cool thing to do was to have toolbars in your application. I also got rid of some other "chrome" so that I have a simple, productive interface. Add these to your .emacs file to get rid of the toolbar and scroll bars, but to keep your menu available (on OS X, it'll be hidden unless you mouse to the top of the screen anyway):   (if (fboundp 'scroll-bar-mode) (scroll-bar-mode -1))(if (fboundp 'tool-bar-mode) (tool-bar-mode -1))(if (fboundp 'menu-bar-mode) (menu-bar-mode 1))   Org Mode   I pretty much live in Org Mode. It is my go-to environment for authoring documents, keeping notes, making to-do lists and 90% of everything else I do. Org was originally conceived as a combination note-taking and to-do list utility by a fellow who is a laptop-in-meetings sort. I am against use of laptops in meetings and don't do it myself, so my use cases are a little different than his. For me, Org is primarily a way to handle all manner of content within a structure. There are heads and subheads, etc., in Org Mode, and they function like an outline. Org allows you to expand or hide the contents of the tree and also to rearrange the tree. This fits how I think very nicely and I find it to be just a pleasure to use in this way.   Org Mode also has a lot of little things that make life pleasant. For example, the footnote handling is excellent and the LaTeX/PDF output is great. Org has the ability to generate agendas based on the to-do's in all your documents and a nice way to relate them to dates/times. I don't use this for any sort of external commitments, which are handled on a shared calendar, but for creating things and keeping track of what I need to create in the future, it's invaluable. Installing it is as easy as adding the org-mode.el to your Lisp directory and adding these lines to your .emacs, if you want it to indent based on tree location and to open documents fully expanded:   ;; set up org mode(setq org-startup-indented t)(setq org-startup-folded "showall")(setq org-directory "~/org")   The last line is there so that Org knows where to look for files to include in agendas and some other things. I keep Org right in my home directory, i.e., a symlink to the directory that lives in Dropbox, as described earlier.   I have a stuff.org file that is always open in a buffer. I use it like a notepad. Org makes it easy to extract things like TODOs and stuff with deadlines. It's especially useful when you can inline Lisp code and evaluate it whenever you need. Having code with content is super handy. Again, you have access to the actual computer with Emacs, and this is a liberation.   Publishing with Org Mode   I care about the appearance and formatting of my documents. I started my career as a designer, and I think information can and should be presented clearly and beautifully. Org has great support for generating PDFs via LaTeX, which has a bit of its own learning curve, but doing simple things is pretty easy.   If you want to use fonts and styles other than the typical LaTeX ones, you've got a few things to do. First, you'll want XeLaTeX so you can use normal system fonts rather than LaTeX specific fonts. Next, you'll want to add this to .emacs:   (setq org-latex-pdf-process '("xelatex -interaction nonstopmode %f" "xelatex -interaction nonstopmode %f"))   I put this right at the end of my Org section of .emacs to keep things tidy. This will allow you to use more formatting options when publishing from Org. For example, I often use:   #+LaTeX_HEADER: usepackage{fontspec}#+LATEX_HEADER: setmonofont[Scale=0.9]{Input Mono}#+LATEX_HEADER: setromanfont{Maison Neue}#+LATEX_HEADER: linespread{1.5}#+LATEX_HEADER: usepackage[margin=1.25in]{geometry}#+TITLE: Document Title Here   These simply go somewhere in your .org file. Our corporate font for body copy is Maison Neue, but you can put whatever is appropriate here. I strongly discourage the use of Maison Neue. It’s a terrible font and no one should ever use it.   This file is an example of PDF output using these settings. This is what out-of-the-box LaTeX always looks like. It's fine I suppose, but the fonts are boring and a little odd. Also, if you use the standard format, people will assume they are reading something that is or pretends to be an academic paper. You've been warned.   Ace Jump Mode   This is more of a gem than a major feature, but you want it. It works a bit like Jef Raskin's Leap feature from days gone by.9 The way it works is you type C-c C-SPC and then type the first letter of the word you want to jump to. It highlights all occurrences of words with that initial character, replacing it with a letter of the alphabet. You simply type the letter of the alphabet for the location you want and your cursor jumps to it. I find myself using this as often as the more typical nav keys or search. Download the .el to your Lisp directory and put this in your .emacs:   ;; set up ace-jump-mode(add-to-list 'load-path "which-folder-ace-jump-mode-file-in/")(require 'ace-jump-mode)(define-key global-map (kbd "C-c C-SPC" ) 'ace-jump-mode)   More Later That's enough for one post—this may get you somewhere you'd like to be. I'd love to hear about your uses for Emacs aside from programming (or for programming!) and whether this was useful at all. There are likely some boneheaded PHBisms in how I use Emacs, and if you want to point them out, I'd appreciate it. I'll probably write some updates over time to introduce additional features or modes. I'll certainly show you how to use Fugue with Emacs and Ludwig-mode as we evolve it into something more useful than code highlighting. Send your thoughts to @fugueHQ on Twitter.   Footnotes ^ If you are now a PHB of some sort, but were never technical, Emacs likely isn’t for you. There may be a handful of folks for whom Emacs will form a path into the more technical aspects of computing, but this is probably a small population. It’s helpful to know how to use a Unix or Windows terminal, to have edited a dotfile or two, and to have written some code at some point in your life for Emacs to make much sense. ^ http://archive.wired.com/wired/archive/2.08/tufte.html ^ I mainly use this to perform calculations while writing. For example, I was writing an offer letter to a new employee and wanted to calculate how many options to include in the offer. Since I have a variable defined in my .emacs for outstanding-shares, I can simply type M-: (* .001 outstanding-shares) and get a tenth of a point without opening a calculator or spreadsheet. I keep lots of numbers in variables like this so I can avoid context switching. ^ The missing piece of this is the web. There is an Emacs web browser called eww that will allow you to browse in Emacs. I actually use this, as it is both a great ad-blocker and removes most of the poor choices in readability from the web designer's hands. It's a bit like Reading Mode in Safari. Unfortunately, most websites have lots of annoying cruft and navigation that translates poorly into text. ^ Usability is often confused with learnability. Learnability is how difficult it is to learn a tool. Usability is how useful the tool is. Often, these are at odds, such as with the mouse and menus. Menus are highly learnable, but have poor usability, so there have been keyboard shortcuts from the earliest days. Raskin was right on many points where he was ignored about GUIs in general. Now, OSes are putting things like decent search onto a keyboard shortcut. On OS X and Windows, my default method of navigation is search. Ubuntu's search is badly broken, as is the rest of its GUI. ^ AWS S3 has effectively solved file storage for as long as we have the Internet. Trillions of objects are stored in S3 and they've never lost one of them. Most every service out there that offers cloud storage is built on S3 or imitates it. No one has the scale of S3, so I keep important stuff there, via Dropbox. ^ By now, you might be thinking "what is it with this guy and bicycles?" ... I love them on every level. They are the most mechanically efficient form of transportation ever invented. They can be objects of real beauty. And, with some care, they can last a lifetime. I had Rivendell Bicycle Works build a frame for me back in 2001 and it still makes me happy every time I look at it. Bicycles and UNIX are the two best inventions I've interacted with. Well, they and Emacs. ^ This is not a tutorial for Emacs. It comes with one and it's excellent. I do walk through some of the things that I find most important to getting a useful Emacs setup, but this is not a replacement in any way. ^ Jef Raskin designed the Canon Cat computer in the 1980s after falling out with Steve Jobs on the Macintosh project, which he originally led. The Cat had a document-centric interface (as all computers should) and used the keyboard in innovative ways that you can now imitate with Emacs. If I could have a modern, powerful Cat with a giant high-res screen and Unix underneath, I'd trade my Mac for it right away. https://youtu.be/o_TlE_U_X3c?t=19s   Related Posts Your Beautiful Baby VPC on AWS: Part 3 October 16th, 2013   Your Beautiful Baby VPC on AWS: Part 2 October 2nd, 2013

    Read More
  • Python Mocking 101: Fake It Before You Make It

    Welcome to a guide to the basics of mocking in Python. It was borne out of my need to test some code that used a lot of network services and my experience with GoMock, which showed me how powerful mocking can be when done correctly (thanks, Tyler). I'll begin with a philosophical discussion about mocking because good mocking requires a different mindset than good development. Development is about making things, while mocking is about faking things. This may seem obvious, but the "faking it" aspect of mocking tests runs deep, and understanding this completely changes how one looks at testing. After that, we'll look into the mocking tools that Python provides, and then we'll finish up with a full example.   Mocking can be difficult to understand. When I'm testing code that I've written, I want to see whether the code does what it's supposed to do from end-to-end. I usually start thinking about a functional, integrated test, where I enter realistic input and get realistic output. I access every real system that my code uses to make sure the interactions between those systems are working properly, using real objects and real API calls. While these kinds of tests are essential to verify that complex systems are interworking well, they are not what we want from unit tests.   Unit tests are about testing the outermost layer of the code. Integration tests are necessary, but the automated unit tests we run should not reach that depth of systems interaction. This means that any API calls in the function we're testing can and should be mocked out. We should replace any nontrivial API call or object creation with a mock call or object. This allows us to avoid unnecessary resource usage, simplify the instantiation of our tests, and reduce their running time. Think of testing a function that accesses an external HTTP API. Rather than ensuring that a test server is available to send the correct responses, we can mock the HTTP library and replace all the HTTP calls with mock calls. This reduces test complexity and dependencies, and gives us precise control over what the HTTP library returns, which may be difficult to accomplish otherwise.   What do we mean by mocking?   The term mocking is thrown around a lot, but this document uses the following definition:   "The replacement of one or more function calls or objects with mock calls or objects"   A mock function call returns a predefined value immediately, without doing any work. A mock object's attributes and methods are similarly defined entirely in the test, without creating the real object or doing any work. The fact that the writer of the test can define the return values of each function call gives him or her a tremendous amount of power when testing, but it also means that s/he needs to do some foundational work to get everything set up properly.   In Python, mocking is accomplished through the unittest.mock module. The module contains a number of useful classes and functions, the most important of which are the patch function (as decorator and context manager) and the MagicMock class. Mocking in Python is largely accomplished through the use of these two powerful components.   What do we NOT mean by mocking?   Developers use a lot of "mock" objects or modules, which are fully functional local replacements for networked services and APIs. For example, the moto library is a mock boto library that captures all boto API calls and processes them locally. While these mocks allow developers to test external APIs locally, they still require the creation of real objects. This is not the kind of mocking covered in this document. This document is specifically about using MagicMock objects to fully manage the control flow of the function under test, which allows for easy testing of failures and exception handling.   How do we mock in Python?   Mocking in Python is done by using patch to hijack an API function or object creation call. When patch intercepts a call, it returns a MagicMock object by default. By setting properties on the MagicMock object, you can mock the API call to return any value you want or raise an Exception.   The overall procedure is as follows:   Write the test as if you were using real external APIs. In the function under test, determine which API calls need to be mocked out; this should be a small number. In the test function, patch the API calls. Set up the MagicMock object responses. Run your test.   If your test passes, you're done. If not, you might have an error in the function under test, or you might have set up your MagicMock response incorrectly. Next, we'll go into more detail about the tools that you use to create and configure mocks.   patch import unittest from unittest.mock import patch   patch can be used as a decorator to the test function, taking a string naming the function that will be patched as an argument. In order for patch to locate the function to be patched, it must be specified using its fully qualified name, which may not be what you expect. If a class is imported using a from module import ClassA statement, ClassA becomes part of the namespace of the module into which it is imported.   For example, if a class is imported in the module my_module.py as follows:   [in my_module.py] from module import ClassA   It must be patched as @patch(my_module.ClassA), rather than @patch(module.ClassA), due to the semantics of the from ... import ... statement, which imports classes and functions into the current namespace.   Typically patch is used to patch an external API call or any other time- or resource-intensive function call or object creation. You should only be patching a few callables per test. If you find yourself trying patch more than a handful of times, consider refactoring your test or the function you're testing.   Using the patch decorator will automatically send a positional argument to the function you're decorating (i.e., your test function). When patching multiple functions, the decorator closest to the function being decorated is called first, so it will create the first positional argument.   @patch('module.ClassB')@patch('module.functionA')def test_some_func(self, mock_A, mock_B): ...   By default, these arguments are instances of MagicMock, which is unittest.mock's default mocking object. You can define the behavior of the patched function by setting attributes on the returned MagicMock instance.   MagicMock   MagicMock objects provide a simple mocking interface that allows you to set the return value or other behavior of the function or object creation call that you patched. This allows you to fully define the behavior of the call and avoid creating real objects, which can be onerous. For example, if we're patching a call to requests.get, an HTTP library call, we can define a response to that call that will be returned when the API call is made in the function under test, rather than ensuring that a test server is available to return the desired response.   The two most important attributes of a MagicMock instance are return_value and side_effect, both of which allow us to define the return behavior of the patched call.   return_value   The return_value attribute on the MagicMock instance passed into your test function allows you to choose what the patched callable returns. In most cases, you'll want to return a mock version of what the callable would normally return. This can be JSON, an iterable, a value, an instance of the real response object, a MagicMock pretending to be the response object, or just about anything else. When patching objects, the patched call is the object creation call, so the return_value of the MagicMock should be a mock object, which could be another MagicMock. If the code you're testing is Pythonic and does duck typing rather than explicit typing, using a MagicMock as a response object can be convenient. Rather than going through the trouble of creating a real instance of a class, you can define arbitrary attribute key-value pairs in the MagicMock constructor and they will be automatically applied to the instance.   [in test_my_module]@patch('external_module.api_call')def test_some_func(self, mock_api_call): mock_api_call.return_value = MagicMock(status_code=200, response=json.dumps({'key':'value'})) my_module.some_func()[in my_module]import external_moduledef some_func(): response = external_module.api_call() #normally returns a Response object, but now returns a MagicMock#response == mock_api_call.return_value == MagicMock(status_code=200, response=json.dumps({'key':'value'}))   Note that the argument passed to test_some_func, i.e., mock_api_call, is a MagicMock and we are setting return_value to another MagicMock. When mocking, everything is a MagicMock.   Speccing a MagicMock   While a MagicMock’s flexibility is convenient for quickly mocking classes with complex requirements, it can also be a downside. By default, MagicMocks act like they have any attribute, even attributes that you don’t want them to have. In the example above, we return a MagicMock object instead of a Response object. However, say we had made a mistake in the patch call and patched a function that was supposed to return a Request object instead of a Response object. The MagicMock we return will still act like it has all of the attributes of the Request object, even though we meant for it to model a Response object. This can lead to confusing testing errors and incorrect test behavior.   The solution to this is to spec the MagicMock when creating it, using the spec keyword argument: MagicMock(spec=Response). This creates a MagicMock that will only allow access to attributes and methods that are in the class from which the MagicMock is specced. Attempting to access an attribute not in the originating object will raise an AttributeError, just like the real object would. A simple example is:   m = MagicMock()m.foo() #no error raised# Response objects have a status_code attributem = MagicMock(spec=Response, status_code=200, response=json.dumps({‘key’:’value’}))m.foo() #raises AttributeErrorm.status_code #no error raised   side_effect   Sometimes you'll want to test that your function correctly handles an exception, or that multiple calls of the function you're patching are handled correctly. You can do that using side_effect. Setting side_effect to an exception raises that exception immediately when the patched function is called.   Setting side_effect to an iterable will return the next item from the iterable each time the patched function is called. Setting side_effect to any other value will return that value.   [in test_my_module]@patch('external_module.api_call')def test_some_func(self, mock_api_call): mock_api_call.side_effect = SomeException() my_module.some_func()[in my_module]def some_func(): try: external_module.api_call() except SomeException: print(“SomeException caught!”) # this code is executed except SomeOtherException: print(“SomeOtherException caught!”) # not executed[in test_my_module]@patch('external_module.api_call')def test_some_func(self, mock_api_call): mock_api_call.side_effect = [0, 1] my_module.some_func()[in my_module]def some_func(): rv0 = external_module.api_call() # rv0 == 0 rv1 = external_module.api_call() # rv1 == 1   assert_called_with   assert_called_with asserts that the patched function was called with the arguments specified as arguments to assert_called_with.   [inside some_func]someAPI.API_call(foo, bar='baz')[inside test_some_func]some_func()mock_api_call.assert_called_with(foo, bar='baz')   A full example   In this example, I'm testing a retry function on Client.update. This means that the API calls in update will be made twice, which is a great time to use MagicMock.side_effect.   The full code of the example is here:   import unittestfrom unittest.mock import patchclass TestClient(unittest.TestCase):def setUp(self): self.vars_client = VarsClient()@patch('pyvars.vars_client.VarsClient.get')@patch('requests.post')def test_update_retry_works_eventually(self, mock_post, mock_get): mock_get.side_effect = [ VarsResponse(), VarsResponse()] mock_post.side_effect = [ requests.ConnectionError('Test error'), MagicMock(status_code=200, headers={'content-type':"application/json"}, text=json.dumps({'status':True})) ] response = self.vars_client.update('test', '0') self.assertEqual(response, response)@patch('pyvars.vars_client.VarsClient.get')@patch('requests.post') def test_update_retry_works_eventually(self, mock_post, mock_get):   I'm patching two calls in the function under test (pyvars.vars_client.VarsClient.update), one to VarsClient.get and one to requests.post. Since I'm patching two calls, I get two arguments to my test function, which I've called mock_post and mock_get. These are both MagicMock objects. In their default state, they don't do much. We need to assign some response behaviors to them.   mock_get.side_effect = [ VarsResponse(), VarsResponse()]mock_post.side_effect = [ requests.ConnectionError('Test error'), MagicMock(status_code=200, headers={'content-type':"application/json"}, text=json.dumps({'status':True}))]   This tests to make sure a retry facility works eventually, so I'll be calling update multiple times, and making multiple calls to VarsClient.get and requests.post.   Here I set up the side_effects that I want. I want all the calls to VarsClient.get to work (returning an empty VarsResponse is fine for this test), the first call to requests.post to fail with an exception, and the second call to requests.post to work. This kind of fine-grained control over behavior is only possible through mocking.   response = self.vars_client.update('test', '0')self.assertEqual(response, response)   Once I've set up the side_effects, the rest of the test is straightforward. The behavior is: the first call to requests.post fails, so the retry facility wrapping VarsClient.update should catch the error, and everything should work the second time. This behavior can be further verified by checking the call history of mock_get and mock_post.   Conclusion   Using mock objects correctly goes against our intuition to make tests as real and thorough as possible, but doing so gives us the ability to write self-contained tests that run quickly, with no dependencies. It gives us the power to test exception handling and edge cases that would otherwise be impossible to test. Most importantly, it gives us the freedom to focus our test efforts on the functionality of our code, rather than our ability to set up a test environment. By concentrating on testing what’s important, we can improve test coverage and increase the reliability of our code, which is why we test in the first place.   Documentation Links https://docs.python.org/3/library/unittest.mock.html   And, check out fugue.co.   Related Posts Managing Secrets at Scale at Velocity EU January 12th, 2016 AWS Lambda and the Evolution of the Cloud February 1st, 2016 We're now Fugue and we raised $20M January 14th, 2016

    Read More
  • Diagnosing and Fixing Memory Leaks in Python

    Fugue uses Python extensively throughout the Conductor and in our support tools, due to its ease-of-use, extensive package library, and powerful language tools. One thing we've learned from building complex software for the cloud is that a language is only as good as its debugging and profiling tools. Logic errors, CPU spikes, and memory leaks are inevitable, but a good debugger, CPU profiler, and memory profiler can make finding these errors significantly easier and faster, letting our developers get back to creating Fugue’s dynamic cloud orchestration and enforcement system. Let’s look at a case in point.   In the fall, our metrics reported that a Python component of Fugue called the reflector was experiencing random restarts and instability after a few days of uptime. Looking at memory usage showed that the reflector's memory footprint increased monotonically and continuously, indicating a memory leak. tracemalloc, a powerful memory tracking tool in the Python standard library, made it possible to quickly diagnose and fix the leak. We discovered that the memory leak was related to our use of requests, a popular third-party Python HTTP library. Rewriting the component to use urllib from the Python standard library eliminated the memory leak. In this blog, we'll explore the details.   Metrics show the problem: Percentage of total system memory used by the reflector, using the requests library.   Memory Allocation in Python   In most scenarios, there's no need to understand memory management in Python beyond knowing that the interpreter manages memory for you. However, when writing large, complex Python programs with high stability requirements, it’s useful to peek behind the curtain to understand how to write code that interacts well with Python's memory management algorithms. Notably, Python uses reference counting and garbage collection to free memory blocks, and only frees memory to the system when certain internal requirements are met. A pure Python script will never have direct control over memory allocation in the interpreter. If direct control over memory allocation is desired, the interpreter's memory allocation can be bypassed by writing or using an extension. For example, numpy manages memory for large data arrays using its own memory allocator.   Fundamentally, Python is a garbage-collected language that uses reference counting. The interpreter automatically allocates memory for objects as they are created and tracks the number of references to those objects in a data structure associated with the object itself. This memory will be freed when the reference count for those objects reaches zero. In addition, garbage collection will detect cycles and remove objects that are only referenced in cycles. Every byte of memory allocated within the Python interpreter is able to be freed between these two mechanisms, but no claims can be made about memory allocated in extensions.   Python manages its own heap, separate from the system heap. Memory is allocated in the Python interpreter by different methods according to the type of the object to be created. Scalar types, such as integers and floats, use different memory allocation methods than composite types, such as lists, tuples, and dictionaries. In general, memory is allocated on the Python heap in fixed-size blocks, depending on the type. These blocks are organized into pools, which are further organized into arenas. Memory is pre-allocated using arenas, pools, and blocks, which are then used to store data as needed over the course of program’s execution. Since these blocks, pools, and arenas are kept in Python's own heap, freeing a memory block merely marks it as available for future use in the interpreter. Freeing memory in Python does not immediately free the memory at the system level. When an entire arena is marked as free, its memory is released by the Python interpreter and returned to the system. However, this may occur infrequently due to memory fragmentation.   Due to these abstractions, memory usage in Python often exhibits high-water-mark behavior, where peak memory usage determines the memory usage for the remainder of execution, regardless of whether that memory is actively being used. Furthermore, the relationship between memory being "freed" in code and being returned to the system is vague and difficult to predict. These behaviors make completely understanding the memory usage of complex Python programs notoriously difficult.   Memory Profiling Using tracemalloc   tracemalloc is a package included in the Python standard library (as of version 3.4). It provides detailed, block-level traces of memory allocation, including the full traceback to the line where the memory allocation occurred, and statistics for the overall memory behavior of a program. The documentation is available here and provides a good introduction to its capabilities. The original Python Enhancement Proposal (PEP) introducing it also has some insight on its design.   tracemalloc can be used to locate high-memory-usage areas of code in two ways:   looking at cumulative statistics on memory use to identify which object allocations are using the most memory, and tracing execution frames to identify where those objects are allocated in the code.   Module-level Memory Usage   We start by tracing the memory usage of the entire program, so we can identify, at a high level, which objects are using the most memory. This will hopefully provide us with enough insight to know where and how to look more deeply. The following wrapper starts tracing and prints statistics when Ctrl-C is hit: import tracemalloctracemalloc.start(10)try: run_reflector()except: snapshot = tracemalloc.take_snapshot() top_n(25, snapshot, trace_type='filename') tracemalloc.start(10) starts memory tracing, while saving 10 frames of traceback for each entry. The default is 1, but saving more traceback frames is useful if you plan on using tracebacks to locate memory leaks, which will be discussed later. tracemalloc.take_snapshot() takes a snapshot of currently allocated memory in the Python heap. It stores the number of allocated blocks, their size, and tracebacks to identify which lines of code allocated which blocks of memory. Once a snapshot is created, we can compute statistics on memory use, compare snapshots, or save them to analyze later. top_n is a helper function I wrote to pretty print the output from tracemalloc. Here, I ask for the top 25 memory allocations in the snapshot, grouped by filename. After running for a few minutes, the output looks like this: [ Top 25 with filename tracebacks ]197618 blocks 17.02311134338379 MB/Users/mike/.pyenv/versions/3.4.2/lib/python3.4/collections/__init__.py:0: size=17.0 MiB, count=197618, average=90 B105364 blocks 11.34091567993164 MB<frozen importlib._bootstrap>:0: size=11.3 MiB, count=105364, average=113 B60339 blocks 9.233230590820312 MB/Users/mike/.pyenv/versions/3.4.2/lib/python3.4/json/decoder.py:0: size=9455 KiB, count=60339, average=160 B... This shows the cumulative amount of memory allocated by the component over the entire runtime, grouped by filename. At this level of granularity, it's hard to make sense of the results. For instance, the first line shows us that 17 MB of collections objects are created, but this view doesn't provide enough detail for us to know which objects, or where they're being used. A different approach is needed to isolate the problem.   Understanding tracemalloc Output   tracemalloc shows the net memory usage at the time a memory snapshot is taken. When comparing two snapshots, it shows the net memory usage between the two snapshots. If memory is allocated and freed between snapshots, it won't be shown in the output. Therefore, if snapshots are created at the same point in a loop, any memory allocations visible in the differences between two snapshots are contributing to the long-term total amount of memory used, rather than being a temporary allocation made in the course of execution.   In the case of reference cycles that require garbage collection, uncollected cycles are recorded in the output, while collected cycles are not. Any blocks freed by the garbage collector in the time covered by a snapshot will be recorded as freed memory. Therefore, forcing garbage collection with gc.collect() before taking a snapshot will reduce noise in the output.   Per-Iteration Memory Usage   Since we're looking for a memory leak, it's useful to understand how the memory usage of our program changes over time. We can instrument the main loop of the component, to see how much memory is allocated in each iteration, by calling the following method from the main loop: def collect_stats(self): self.snapshots.append(tracemalloc.take_snapshot()) if len(self.snapshots) > 1: stats = self.snapshots[-1].filter_traces(filters).compare_to(self.snapshots[-2], 'filename') for stat in stats[:10]: print("{} new KiB {} total KiB {} new {} total memory blocks: ".format(stat.size_diff/1024, stat.size / 1024, stat.count_diff ,stat.count)) for line in stat.traceback.format(): print(line) This code takes a memory snapshot and saves it, then uses snapshot.compare_to(other_snapshot, group_by='filename') to compare the newest snapshot with the previous snapshot, with results grouped by filename. After a few iterations to warm up memory, the output looks like this: [ Top 5 with filename tracebacks ]190.7421875 new KiB 1356.5634765625 total KiB 1930 new 13574 total memory blocks: (1) File "/Users/mike/.pyenv/versions/3.4.2/lib/python3.4/linecache.py", line 02.1328125 new KiB 12.375 total KiB 32 new 86 total memory blocks: (2) File "/Users/mike/.pyenv/versions/3.4.2/lib/python3.4/tracemalloc.py", line 01.859375 new KiB 18.7001953125 total KiB 3 new 53 total memory blocks: (3) File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 0-1.71875 new KiB 34.5224609375 total KiB -2 new 91 total memory blocks: File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 01.66015625 new KiB 61.662109375 total KiB 18 new 260 total memory blocks: File "/Users/mike/.pyenv/versions/3.4.2/lib/python3.4/urllib/parse.py", line 0 The linecache (1) and tracemalloc (2) allocations are part of the instrumentation, but we can also see some memory allocations made by the requests HTTP package (3) that warrant further investigation. Recall that tracemalloc tracks net memory usage, so these memory allocations are accumulating on each iteration. Although the individual allocations are small and don't jump out as problematic, the memory leak only becomes apparent over the course of a few days, so it's likely to be a case of small losses adding up.   Filtering Snapshots   Now that we have an idea of where to look, we can use tracemalloc's filtering capabilities to show only memory allocations related to the requests package: from tracemalloc import Filter filters = [Filter(inclusive=True, filename_pattern="*requests*")] filtered_stats = snapshot.filter_traces(filters).compare_to(old_snapshot.filter_traces(filters), 'traceback') for stat in stats[:10]: print("{} new KiB {} total KiB {} new {} total memory blocks: ".format(stat.size_diff/1024, stat.size / 1024, stat.count_diff ,stat.count)) for line in stat.traceback.format(): print(line) snapshot.filter_traces() takes a list of Filters to apply to the snapshot. Here, we create a Filter in inclusive mode, so it includes only traces that match the filename_pattern. When inclusive is False, the filter excludes traces that match the filename_pattern. The filename_pattern uses UNIX-style wildcards to match filenames in the traceback. In this example, the wildcards in "requests" match occurrences of "requests" in the middle of a path, such as "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/sessions.py".   We then use compare_to() to compare the results to the previous snapshot. The filtered output is below: 48.7890625 new KiB 373.974609375 total KiB 4 new 1440 total memory blocks: (4) File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/structures.py", line 01.46875 new KiB 16.2939453125 total KiB 2 new 49 total memory blocks: File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests_unixsocket/__init__.py", line 0 -1.4453125 new KiB 34.2802734375 total KiB -2 new 96 total memory blocks: (5) File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/sessions.py", line 0-0.859375 new KiB 31.8505859375 total KiB -1 new 85 total memory blocks: File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 00.6484375 new KiB 20.8330078125 total KiB 1 new 56 total memory blocks: File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 0 With the Filter in place, we can clearly see how requests is using memory. Line (4) shows that roughly 50 KiB of memory is lost in requests on each iteration of the main loop. Note that negative memory allocations, such as (5), are visible in this output. These allocations are freeing memory allocated in previous loop iterations.   Tracking Down Memory Allocations   To determine which uses of requests are leaking memory, we can take a detailed look at where problematic memory allocations occur by calling compare_to() with traceback instead of filename, while using a Filter to narrow down the output: stats = snapshot.filter_traces(filters).compare_to(old_snapshot.filter_traces(filters), 'traceback') This prints 10 frames of traceback (since we started tracing with tracemalloc.start(10)) for each entry in the output, a truncated example of which is below: 5 memory blocks: 4.4921875 KiB File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/sessions.py", line 585 r = adapter.send(request, **kwargs) File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests/sessions.py", line 475 resp = self.send(prep, **send_kwargs) File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests_unixsocket/__init__.py", line 46 return session.request(method=method, url=url, **kwargs) File "/Users/mike/.pyenv/versions/venv/lib/python3.4/site-packages/requests_unixsocket/__init__.py", line 60 return request('post', url, data=data, json=json, **kwargs) The full traceback gives us the ability to trace backwards from memory allocations to the lines in our project code that generate them. In the case of this component, our uses of requests came from an internal storage library that used an HTTP API. Rewriting the library to use urllib directly eliminated the memory leak. Metrics indicate the problem is solved: Percentage of total system memory used by the reflector, after removing requests and switching to urllib.   Memory Profiling: Art or Science?   tracemalloc is a powerful tool for understanding the memory usage of Python programs. It helped us understand module-level memory usage, find out which objects are being allocated the most, and it demonstrated how the reflector’s memory usage changed on a per-iteration basis. It comes with useful filtering tools and gives us the ability to see the full traceback for any memory allocation. Despite all of its features, however, finding memory leaks in Python can still feel like more of an art than a science. Memory profilers give us the ability to see how memory is being used, but oftentimes it’s difficult to find the exact memory allocation that is causing problems. It’s up to us to synthesize the information we get from our tools into a conclusion about the memory behavior of the program, then make a decision about what actions to take from there.   We use virtually every available Python tool (test frameworks, cProfile, etc.) to make Fugue’s system reliable, performant, and easy to maintain. The broker and reflector both take advantage of Python's introspection to make judgments about dynamic calls to the AWS API, which allows us to focus on logic rather than coding exhaustive cases. Fugue leverages the strengths of Python where it makes sense in the system, which ultimately means more product stability and extensibility for end-users.   Related Posts Fugue Computing: Next Generation Infrastructure Automation Is Here August 9th, 2016   Testing Migration: Shifting from Manual to Automatic March 21st, 2016   Python Mocking 101: Fake It Before You Make It February 11th, 2016   Our Functional Future or: How I Learned to Stop Worrying and Love Haskell January 27th, 2016

    Read More