Maintaining MFA-Based AWS Cross-Account Credentials

AWS’ CLI tools provide a powerful set of tools when working with large batch jobs or even simpler day-to-day tasks. When I recently found myself tasked to convert 100+ virtual machine images, totaling over 6TB of data, into AWS AMIs, my first thought was “write a script!”. This ultimately presented a bit of a hurdle in that cross-account credentials expire every hour and must be renewed, especially with long-running processes.

After moving all the images into S3 from a Snowball export, I cooked up a quick shell script to start invoking the ec2-import-image command against the images. The first hiccup occurred upon my discovery that only 20 image import tasks can be running at any one time in an account. Once I had refactored my tooling to accommodate this, it was off to the races… sort of. Since these tasks take a while to run, naturally I wanted to check on their process periodically. The ec2-describe-import-image-tasks command provided some nice output, including overall status as well as a percentage-based indication of the progress. I fed this output into a jq filter, which would at a quick glance give me some nice high-level output regarding batch status (which I’ll cover in a future post). So, after kicking off my first batch and getting a feel for how often to check back, I grabbed some dinner. When I came back to check, I re-ran the describe-import-image-tasks command I ran earlier: creds expired!

Annoyingly, I was forced every hour to refresh my credential tokens. This meant running the sts assume-role command whenever my credentials had expired, copying the output out of the terminal and into my .aws/credentials file, saving the file, and re-running my initial command. Not cool. To circumvent the issue, I wrote a shell script that automates the retrieval of the new credentials and updates your .aws/credentials file for you. The upshot? Now you can chain this command with other commands. It is designed to work with accounts where you are required to use an MFA token, since this wouldn’t be explicitly required if tokens weren’t being used. We have started a repo where we’ll place tools like this moving forward; the script is already in the repo (look in the IAM directory): so head over to our GitHub tools repo, clone it, and make managing your cross-account credentials requiring tokens way easy!