Continuing on from my last post, I mentioned dealing with the feeling of being left behind. Of not being able to keep up anymore. I don’t think “obsolete” was quite the right word for it — it was more like being stuck in a mental hole. The ground shifts faster than you can move, you can’t find your footing, and after a while you stop trying to climb out because you can’t even see the top anymore. A lot of that was driven by stress, burnout, and honestly just not being inspired by the work anymore. When the passion drains out, everything feels harder than it should.

No doubt others have faced this before. Technology changes, the industry moves, and if you’ve been in it long enough, there will be a moment where you look around and think “when did all of this happen?” For me, the thing that really amplified that feeling — that really deepened the hole — was AI.

The Speed of It All

It wasn’t just that AI arrived — it’s the speed at which it hit. It was changing almost weekly. Still is, honestly. Which tools are best, how to use them, what they’re capable of, what’s hype and what’s real — it was a lot to process when you’re already stretched thin with day-to-day work.

I’ll freely admit that eighteen months to two years ago, when this was really starting to take hold, I was skeptical. I didn’t find it particularly useful for what I was doing. I struggled to see where it was going to fit in. I wasn’t writing code on a daily basis anymore, so the code generation side of things didn’t click for me. And I didn’t have the bandwidth — mentally or time-wise — to properly experiment with it.

So I just… didn’t. And watching everyone else seemingly get it while I was still on the sidelines didn’t help with the whole “being left behind” thing.

Changing My Mind

But here’s where I’m at now: AI is well embedded in my workspace, my job, and my company pushes it. And I’ve been able to adapt to it and — more importantly — learn with it.

Whether or not you agree with its direction, whether you want to use it or not, I think it’s pretty clear to everyone at this point that it’s here to stay. It’s proven itself in certain use cases, and it’s getting better at a pace that’s genuinely mind boggling. This isn’t one of those tech trends that comes and goes. It feels different.

Now, I know people who are concerned about this. Colleagues, friends, people inside and outside of tech. There are real concerns — from data privacy, to the articles about AI breaking safeguards, to the big one: “it’ll take my jobs.” I’m not going to tell anyone their concerns aren’t valid. They are. How each of those things plays out would be it’s own discussion.

But I will share where my head landed on it.

Starting Small

The way I approached it was deliberately small. Find a task I already need to do. See if AI can help with it. Learn from that interaction. Repeat.

It sounds simple, but that’s kind of the point. If you try to go too big too fast — generate an entire codebase, for instance — you’re not going to be able to review or understand what comes out the other end. You lose the learning opportunity, and you lose the ability to validate the output.

So I treated it like any other tool. It can be used well, it can be used badly. And the quality of what you get out of it really comes down to your ability to break problems into smaller pieces, ask the right questions, and critically evaluate what comes back.

How I Actually Use It

Here’s where it’s landed in my day-to-day:

Meeting notes and organizing. I record meetings, get transcriptions, and have AI summarise them into action items and key points. These go straight into Obsidian as markdown files — my note-taking app of choice. Then I can ask the tooling to search across those notes later. “What did I discuss with this customer last week?” or “What activities have I committed to?” It’s become a genuine productivity boost for the advisory side of my role.

Research and best practices. When a customer is talking about building an application with specific systems or frameworks, I can quickly get AI to pull together best practices, security considerations, and architectural patterns. Beyond that, it’s been great for generating first drafts of documents, architecture diagrams, and work plans. Things that used to take me half a day to structure from scratch now have a solid starting point in minutes. I still review and reshape everything, but the blank page problem is basically gone. It’s not replacing my knowledge — it’s accelerating the research and preparation phase significantly.

Coding and prototyping. I’ve used it in a few forms here. Working with structured frameworks to help generate requirements, design, and build — both web frontends and infrastructure-as-code (CloudFormation, Terraform). I also spent a day building a playable video game using Claude Code and a Godot framework I found on GitHub. Eight hours, start to finish. That was genuinely fun and something I wouldn’t have attempted without the tooling.

This blog. Even this blog post started as a voice transcript in Obsidian that got cleaned up with AI assistance. Meta, I know.

The key thing after all of that generation work is validation. You still need to read what comes out. Has it phrased things correctly? Has it put the logic together right? Is the code actually doing what you think it is? That human review step isn’t going away.

The Bigger Shift

What AI has actually done for me — and this is probably the more important point — is re-spark my passion for this industry. I’d lost it for a while. The combination of burnout, role changes, and feeling like I couldn’t keep up had drained the curiosity out of me.

But working with these tools, experimenting, building things I wouldn’t have attempted before — it’s reminded me why I got into technology in the first place. Not because I need to write every line of code myself, but because I enjoy solving problems and building things.

It’s also shifted where I see my value. I don’t need to be the one writing the automation or the Terraform anymore. I can think bigger about the problem. What do we actually want to achieve? What’s the real core issue here, not just the technical symptom? Is the thing we’re building actually solving the right problem, or are we band-aiding something?

That strategic thinking, combined with the experience of almost two decades in the trenches, turns out to be pretty useful. AI handles the implementation speed — I bring the context, the critical thinking, and the “wait, have we thought about this?” that only comes from having seen enough things go wrong.

On the “Taking Jobs” Thing

At last year’s re:Invent, Werner Vogels — Amazon’s CTO — confronted this head-on. His message was pretty blunt: yes, you might be left behind if you don’t evolve. And look, that’s a harsh way to put it. It genuinely sucks to hear.

But I think it’s also always been true in tech. There’s always been change. Physical datacentres gave way to virtualisation, which gave way to cloud. Manual deployments gave way to CI/CD pipelines. On-prem mail servers gave way to Exchange Online and Gmail. Sysadmins who refused to learn cloud struggled to stay relevant. Every generation of technology has made the previous one obsolete for someone. This is just the next iteration of that same pattern.

I remember years ago, speaking to a colleague who was a tape backup specialist. He was worried about those systems going away — everything moving to disk and cloud-based backup. I was the cloud guy at the time, and I was honest with him. “Yeah, I think they will. You probably want to learn some other things too.” It felt blunt saying it, but it was true. And now here I am on the receiving end of that same conversation, just with a different technology doing the disrupting. Funny how that works.

That doesn’t make it easy. But it does mean the playbook is the same as it’s always been: stay curious, be willing to learn, and figure out where your existing skills apply to the new landscape.

And look, I’ll be honest — AI is fundamentally changing my job. I’m effectively teaching customers to replace my technical knowledge with various AI services. That’s the reality of it. And I’m OK with that. Because my value now isn’t in being the person who knows the answer — it’s in what we call “seeing around corners.” What does my experience tell me about what we need to find out? How do we validate the technical information we’re getting, particularly for the specific situation or use case at hand? An AI can give you an answer, but it takes someone who’s been through enough projects to know which questions haven’t been asked yet.

There’s also the human element. Providing people with the confidence that they’re heading in the right direction. That reassurance, that sanity check, that “yes, this approach makes sense and here’s why” — that’s something that comes from experience and trust, not from a model. For me, that’s mentoring, problem-solving, and strategic thinking — applied through new tools rather than replaced by them.

What’s Next

A quick caveat — everything above is just my opinion. I don’t consider myself a font of knowledge or authority on this topic. I’m just a guy who’s been in the industry a while, sharing how I’ve navigated this particular shift. Take what’s useful, leave what’s not.

This wasn’t a very technical post, and I know that. I will go into more detail on some of the examples I mentioned — the coding projects, the productivity workflows, the tools I’m using. But for now I wanted to set the context of how I got from “AI is probably overhyped” to “AI is genuinely changing how I work for the better.”

I’m still figuring out the cadence for this blog. It’s not very well planned yet, if I’m being honest. But hopefully I can evolve it into something more structured over time. You’ll hear from me again. We’ll keep going.

Take care, and keep learning.

So I really want to get back into this blog thing. It’s been a while. I’d effectively forgotten, lost confidence in it, and left it for dead. But there are quite a few reasons that happened…

After my last proper update, I joined one of the big cloud hyperscalers. And I’ve been there ever since — almost nine years now. In that time I’ve worn a few different hats, learned a huge amount, and honestly, gone through some pretty significant shifts in how I think about my career and myself.

The Journey Through the Roles

When I first joined, I was in the professional services team. Paid consultant, statements of work, the whole deal. Effectively a cloud architect and engineer, doing hands-on building in customer environments, helping them design solutions, and advising on strategy. The role really ran the gamut — advisory, consulting, and getting stuck in with the actual implementation. It was great. I learned more in those early years than I probably had in the few years before.

From there I moved into an APJ specialty team, which was a really interesting gig. It was twofold — still being brought in as a specialist on customer projects, but also helping train other team members on project delivery tooling and techniques. These were prescribed artifacts, code, processes, and design patterns to help accelerate customer projects. Building the playbook and teaching others to run it. That was genuinely rewarding work.

Unfortunately that team got dismantled, as these things sometimes do in large organisations. So I moved around again and landed in a solutions architect role. Still a really cool position, but significantly less hands-on than what I was used to. Less building directly with customers, more advisory and strategy. That’s been an interesting adjustment, to put it mildly.

When the Spark Fades

Here’s the honest bit, and probably the real reason I’m writing this.

Over the last couple of years, I’ve had to reinvent myself. Maybe not reinvent entirely, but definitely change the way I work or the way I look at my paths. I used to be so focused on being technically adept — doing the code, the automation, the infrastructure builds. That was my identity in a lot of ways. But as the roles changed, and as my experience level and tenure grew, I found myself needing to re-understand where I fit in.

For a while, I genuinely thought I was becoming obsolete. I couldn’t keep up with every new tool and framework the way I used to. The negative thoughts crept in — not good enough, being left behind, this isn’t for me anymore. The work stopped feeling like a passion and started feeling like just a job. And if you’ve been in tech long enough, you probably know how draining that can be.

To be clear, there was no real fault for this on any one thing. Given long enough in any company or industry, and things change. Though I do think a big part of it was rigid expectations on how I wanted things to be. Not being open to other opportunities or ideas. And I got stuck in the drudgery of it all. And when you’re in that headspace, it’s hard to see a way forward.

Doing the Work

So the last couple of years have really been about figuring out how to do this better. Not just the job, but how I approach everything mentally. I’ve done a lot of thinking — and a lot of personal mental health work — around understanding the way I operate. Where my brain gets obsessive about certain topics, where it spirals into negativity, and how to catch those patterns before they take hold.

What I’ve started to realise is that my value isn’t just in the technical execution anymore. It’s in the experience. Almost three decades of building, breaking, fixing, and designing systems gives you a perspective that’s genuinely useful — even if you’re not the one say writing the Terraform for example. There’s also the aspect that I started and developed foundational IT skills early. I learned bits of servers, networks, databases, and other core topics. That set good base knowledge for problem solving.

I’ve also noticed that people have started leaning on me for advice, particularly newer team members. Not because I set out to be a mentor, but just through tenure and being around long enough to have seen a lot of things go wrong (and occasionally go right). And it’s made me think — I can actually do this. Coaching, guiding, sharing what I’ve learned. It’s not something I’d considered as a skill before, but it turns out it might be one of the more valuable things I have to offer.

Finding the Groove

So that’s really where I’m at. Trying to be more present, more positive about the work I do, and more intentional about how I use my skills. Not just the technical ones, but the softer stuff — communication, mentoring, strategic thinking. The things that come from years of being in the trenches.

I’m also trying to restart my curiosity. AI is obviously the big thing right now, and I want to understand how it affects what I do and how it can be utilised in my work. But beyond that, I want to find those spaces that reinspire me. To kick off the spark that I once had about technology and figure out how to bring that to life again. Maybe even combine some of the things I’m passionate about outside of work into what I do professionally.

This is more than just a work thing, really. It’s about life in general. Finding my groove again after spending years heads-down in implementation and advisory without coming up for air often enough.

What This Blog Becomes

Now, this might not always be the strictly DevOps content that this blog originally started with. I’m going to keep the name — Driven by DevOps still feels right, even if the scope expands a bit. I want to write about things I’m working on, things I’m coming across, thoughts on the industry, career stuff, and whatever else feels worth sharing.

I’m also putting myself out there a bit more, which doesn’t come naturally to me. It never really has. But I’ve learned that staying quiet and heads-down isn’t serving me the way it used to. So here we are.

Hopefully this sets the tone for where things are heading. Hopefully I can bring more of these to the table and offer something useful to others who might be in a similar spot. And hopefully — though I’ve learned not to make promises — the posts will come a bit more regularly this time around.

We’ll see how it goes. Thanks for sticking around, and I’ll talk to you soon.

I’ve been working on a project recently, and it may come across as counter-intuitive, but it has involved creating a Puppet environment in AWS. This is a bit different to the way I would normally work with an AWS deployment, but the future of the project will end up being onsite as well as cloud, and involves Windows as well as Linux. The customer manages these as long life servers rather than a distinct dynamic environment.

In either case this was a large greenfield project with the endgame being a continuous delivery system for a middleware stack. If I was to do a blog on the entire thing, it would probably take 4 months to complete (Though this blog entry has taken that long anyway). So I have pulled out a short interesting part.

An existing Puppet Environment

I’m not going to go into the setup of a Puppet server, that’s for another time, so we are going to base the examples on the fact an existing Puppet Enterprise server exists and DNS works, and we do have a Puppet control-repo correctly configured. This can work with the Puppet Open Source server as well.

Now what we wanted to achieve here is the creation of AWS instances that will automatically register with the Puppet Master and run their configuration with, as you should expect, zero input from admins. The old process would have involved logging in to the newly created server, downloading the puppet agent and config, and installing it, or a script at best. We can do better than that.

The Puppet Master needs to store agent binaries

To make setup of the agent easier we can store binaries of the agent installers for any OS or OS version that we are going to need. This can actually be done in the Puppet Master. We log in to the Puppet Enterprise console, and go to the Classification section. There will be some pre-defined groups done by the Puppet Enterprise installation. Expand “All Nodes”, then expand “PE Infrastructure”. Select the item “PE Master”. Now you are in the group definition, select the “classes” tab, which then should look similar to the following example:

At this point you will see a list of classes already assigned to this group. In the “Add new class” textbox type ‘pe_repo::platform’, this should case an autocomplete popup to appear listing the possible classes:

Find the entry for the OS and version you require, select it, and click “Add class”. At this point the Puppet Console will ask you to confirm changes. Once they are confirmed Puppet will run through the required processes, and download the agents, storing them locally.

Node Classification

Now whether we are assigning a role to a server via fact or we are using an External Node Classifier (ENC), a puppet node needs be to classified, so the server can determine what configuration manifests to generate and provide to the puppet agent. ( This step does assume you have configured branches in your control-repo for environments correctly). In this situation we are using Groups in the Puppet Enterprise console. A group has been created for each environment, with subgroups being created for each server role or type. The groups use facts to determine which servers belong in which group.

We login to our Puppet Management Console and go to the Node Management - Classification section. Click “Add Group” and input the required details. To make it easy the Group name and Environment should match the environment name used in the UserData example further on.

Once the group is created we click on it and start on the rules to classify nodes:

In the rules tab, we start by typing the fact name that we want in the the Textbox below fact. In this case it is “agent_specified_environment” as the agent is going to tell us what it belongs to. Then we select the Operator, with standard options like must equals, does not equal, and some various like options. Then we input the value we want the fact to be (or not be). Then click “Add Rule”. And apply the changes. At this point we can go to the classes tab to add items to this group, or create Subgroups for further detailed classification. The classes to add would be things like roles and profiles, which should be defined to do configuration on the agent for us.

Creating the Agent Node

Now we will move on to the actual creation of an EC2 instance and the configuration of the puppet agent. However you choose to create EC2 instances should work, as long as you have access to provide user_data. That is the key. To clarify, user_data is a set of commands or script that is run when the the instance starts up.

Most of my work is usually done in Ansible, but here I will show you a snippet of cloudformation template user_data:

UserData:
   Fn::Base64: !Sub |
      #!/bin/bash
      echo ${hostname}.localdomain > /etc/hostname
      sed -i '/HOSTNAME/c\HOSTNAME=${hostname}.localdomain' /etc/sysconfig/network
      echo preserve_hostname=true >> /etc/cloud/cloud.cfg
      curl -k https://puppet.localdomain:8140/packages/current/install.bash | sudo bash -s main:certname=${hostname}.localdomain main:server=puppet.localdomain custom_attributes:challengePassword=${puppet_challenge_password}
      systemctl stop puppet
      echo "environment = ${environment}" >> /etc/puppetlabs/puppet/puppet.conf
      echo -e "[agent]\npluginsync = true" >> puppet.conf
      mkdir -p /etc/puppetlabs/facter/facts.d
      echo "service=${service}" >> /etc/puppetlabs/facter/facts.d/facts.txt
      systemctl enable puppet
      sudo reboot

And to explain what is happening here: Lines 4,5 and 6 deal with setting the hostname of the server, and making it stick during start and stop processes. Usually I wouldn’t do this for dynamic EC2 instances, but for this environment it was required.

Line 7 downloads the puppet agent install script from the Puppet Master server and then runs it.

• main:certname defines the name used in the certificate the puppet agent generates. The cert is what the agent and master use to identify it is a valid agent.
• main:server defines the puppet master server fqdn the agent will connect to.
• custom_attributes:challengePassword is a password configured in the master, that the agent will use to authenticate its registration.

Line 8 stops the puppet agent allowing us to do further config.

Line 9 sets the environment the agent will define itself as. This is used to help the Puppet Master classify the agent automatically.

Line 11 and 12, Sets custom facts for puppet to utilise. These facts are loaded by the agent and presented to the Master. They can then be used to further classify the agent, or be used as variables in manifest execution.

The new agent is then rebooted so ensure the new hostname and config are set.

Autosign or Challenge Password?

When we register agents there are a few options we can use. By default Puppet waits for an Admin to login and approve the registration. Thats far too manual however. We can also autosign registrations, so agents get added automatically, but we probably don’t want every agent that gets installed to add itself to the inventory. Every agent uses a puppet node entry from the license, either using them up, or costing more money. There is an option with autosign.conf to only approve agents with certain names in their fqdn. But that could still mean more registrations than we actuall want.

The last option is a challenge password. This blog explains the option quit well: https://danieldreier.github.io/autosign/. The process involes installing the correct gem on the Puppet Master and running through the config. During the process we specify a password. This is logged in the configuration, and then must be used when the Puppet agent creates its cert request (As shown in the userdata example above). The password allows the agent to authenticate with the Master. Allowing us to make our configuration process a little more secure.

Off and Running

Going through all those details above, we can now deploy an EC2 Instance and know that it will be configured without any hands on work. And it will continue to be managed by Puppet. Deployment and Config management handled automatically. That’s the whole premise of DevOps isn’t it. Hope you are all doing well. Hopefully the next blog post will be sooner rather than later.

So it’s been a while since my last update. I’ve been super busy on customer projects and deployments. There’s been some Architecture design and well as large scale implementation projects.

But the main awesome news and activities of the last month has been being able to participate in Amazon Web Services re:Invent 2016. It is the largest annual convention for AWS and takes place in Las Vegas, Nevada. It is full of keynote announcements, training sessions, hands-on labs, certification exams, and a vendor expo. It takes place over the course of a week, though the main conference is considered to be 2 days of the Wednesday and Thursday. Now many technical people across the globe will have blogged about all the specific technical announcements. I’ll hit some of them here, but the main point of this blog entry will be to try and describe the overall experience for someone who hasn’t been before. And maybe some tips to help navigate the chaos that is Vegas and re:Invent.

Planning what to see

Amazon always announces most of the courses and talks in a good timeframe before re:Invent is to take place. Much of the pre-set agenda is paid training courses. These are often quite good, but should be studied carefully as your choices should really take into account your experience level or job position, to get the best value out of them. Some are business oriented, and arranged more for project managers or sales partner teams. Others are much more technical, for AWS newbies, or very experienced users (from System Administrators to Developers).

Go through the schedule, find your budget, and book the courses you think will be good. They do fill up relatively fast. These courses are generally in high demand, so be quick.

Arrive the weekend before

To be able to take in the full amount of options and activities during the week, arrive early. The weekend before is a good idea. If you’re in a position like I am of being a consultant, it might be a combined training, sales, networking, certification week. It was going to be full-on. So we arrived in on the Sunday, and were able to get settled in about midday. So the afternoon was able to be about getting our bearings and sorting out where we needed to be an when. Fortunately our hotel was directly across the road from the main location for the conference. So a 5 minute walk and we could be there. Depending on where you are it could be a walk, or a taxi/uber ride.

A good plan if you get there in that weekend, is to get registration done on Sunday afternoon/evening. They open the venue and allow early registration pickup. Best to head along to this unless you want to be up super early on Monday. Your registration will get you the first bunch of swag. AWS participants this year got a nice hoodie, and an Amazon Echo Dot! Thats a mini version of the Echo running Alexa. About the size of a hockey puck, and should be interesting to play with (I haven’t really started yet)

Sunday evening we already had client dinners booked. So we were somewhat off and running getting business done.

The First 2 days

Monday and Tuesday was when my booked training sessions would be happening, and it’s best to get important training done on these 2 days, the rest of the week becomes more about announcements and tech. The sessions on these days may be part or full day events, running from 9 to 5, with a lunch break. Mine were full day events, so after grabbing breakfast I headed off for the courses.

My first session was the DevOps Certification Bootcamp. This is designed to run you through the items required for the AWS DevOps Certification. You are given an environment running CodeDeploy, CodePipeline and with access to a code respository. As the class progresses you perform lab tasks against this environment. Deploying a new application, upgrading the application, and performing a blue/green deployment between the 2 you now have. This is an excellent insight into these services and how to use them. I expect to do more study before I take the DevOps exam, but this was a good start.

Tuesdays session was the Cloud Transformation training. This is not a technical session, but more Project management or Project Lead oriented course. It was designed to run you through the process of arranging your organisation in readiness for moving your technology or I.T services to cloud providers. From organisational structure, to hiring skillsets, and identifying priorities and possible complications. I wasn’t sure this would be my thing, but being a consultant now and AWS partner, this ended up being quite useful. It’s likely I will end up needing to use the knowledge that came across here for future projects.

Just a quick note on meals. AWS provides a breakfast and lunch each day. A area of the location is set aside for the many many tables required to feed everyone. Try and get to breakfast around 7-7.30. It’s just before it gets busy.

Tuesday Night

After the important business of training, Tuesday night starts off the interesting events and probably introduces the stuff that might get you excited about re:Invent. First off, the Expo portion opens for an intro night. Vendors market their wares, other stands show future items coming, but it ends up being a mad rush for free swag. Tshirts were given away almost everywhere, and other cool items like drones were being raffled off. The only drawback here is that all the vendors will need to scan your badge, this is where the spam starts. If you are organised that’s not a big deal. It also may allow you to seen vendors that may be useful or interesting for you. I came across Fugue, Zerto, and Cloudcheckr (Though I use them already).

Also on Tuesday night is something that should not be missed. “Tuesday Night Live with James Hamilton”, as the title give it away it is presented by James Hamilton, a AWS VP and Distinguished Engineer. It’s a more technical session and he often goes through how AWS designs, implements, and operates at the scale it does. Providing stats about the insane growth they face every year. This year it involved a team from NASA describing how they use AWS for data analysis. As always re:Invent sessions are streamed and available online after the fact. Check it out if you are interested…

Wednesday

This starts re:Invent proper. The first item of the day (after breakfast at least), is the Keynote presented by Andy Jassy. The first day keynote starts of general and dives into infrastructure and platform services additions. New Compute instance sizes, storage options, new AI based services, and IoT options. Each item will be interesting to different people in different ways, but that’s just due to the large breadth of things you are able to do on the AWS platform.

Thursday morning also starts of with a Keynote, this time presented by Werner Vogels, the CTO Engineer Evangelist extraordinaire of AWS. This presentation went into how important strategy and agility is, not just for AWS, but for it’s customers as well, and really pushed the fast growing “Serverless” architecture movement. Then Werner went into more technical announcements of products, services, and features. A few items peaked my interest, Elastic GPUs, and AWS Systems Manager. At the end of his keynote Werner also always announces who the DJ is for the re:Play party, this year being Martin Garrix.

An important thing to note is as the keynotes happen, AWS releases updates to the agenda for the week. New talks about the services just announced will appear, keep tabs on the agenda and book the items you want. They will fill up very fast. The rest of these 2 days are open with many other sessions, labs, and certification opportunities. I was able to complete my SysOps certification on Wednesday, and took hands-on lab sessions on Thursday.

Thursday night re:Play!

Then there is the party. re:Play takes place in the evening after the rest of the day has finished. It’s huge, and has all kinds of things to do. Not only is there a big dance floor for the generally epic DJ sessions, but the bars are open all night, and there are many activities. This year there was a climbing wall, laser maze, dodgeball arena, and human foosball field. This is a great time to relax, most of the week is done, take it easy, mingle, and really try and take in the atmosphere.

Friday also continues the agenda with talks, but stops around 1pm. And depending on how much you got up to at the party, you may not want to attempt to much.

Networking

A large part of the overall convention is the networking opportunities. Whether it’s during lunch breaks, in sessions, or at the expo, you will have the chance to talk to other AWS employees and users. Everyone is there for the same reason, and at meals tables are often shared and you can strike up conversation with people from all over the globe. Sometimes in sessions you may be put into teams as well, which will often lead to you learning different perspectives.

Overall

As I had not been to a re:Invent before I thought this was a great experience, and highly recommend anyone go if they are using AWS. It may open up more possibilities on how you use the platform. You will meet many people, and be introduced to options you may not have seen before. Being it is Las Vegas it can be a bit of a sensory overload, it’s busy and there is constant light and noise all of the time. Make sure you organise leisure time every so often during the week (it’s Vegas, so there is plenty of non-work activities to break and do). I’ll leave you with an image of my downtime activities. I’m a speed freak so it was awesome. I will be back soon with more technical posts again. Hope you all are well, Christmas is coming, so make sure you take a break.

One thing that has become common in AWS usage is Assumed Roles. As an IAM user (or even other services), you are able to switch (or assume) roles in the AWS account, allowing you to change to a different set of access privileges. This becomes very useful when you are running multiple accounts. All IAM users can be setup in one account, but have the ability to assume roles in the other account(s). This means user credentials only need to be managed in one place, making overall admin more simple. Much of this is better explained in Amazon Docs or Other Blogs

Problem in Existing Automation:

Recently my company created some new AWS accounts and decided to move the lab environment to a new one. This meant we would no longer access the lab with direct credentials, but via an assumed role, with the included complexity of requiring MFA. I have a large repository of Ansible scripts, and unfortunately, they were designed to use default credentials setup in my AWS CLI. I have been working on solving this problem, and enabling my scripts to work with or without an assumed role based on variables I configure in files.

Setting up AWS CLI:

The first step is getting profiles setup in your AWS CLI config. This allows you to have multiple credentials ready for use in different accounts, but also allows profiles for assumed roles. In this example, my default profile is the main account where my user is (know as the jump account), and the lab profile is the account I will be assuming a role into (the destination account). The access key and secret key would be configured in the ~/.aws/credentials file under the profile [default]. The ~/.aws/config file should look like this:

The <assumed role> is the name of the role you need to be in the destination account. You or your AWS admin should have already set it up with the required permissions. The mfa_serial line is the value for the MFA device you have configured. It can be found in your user page in the AWS console. Now to test if that works. Grab a shell, and run (Enter an MFA token if you have that setup):

$ aws ec2 describe-instances --profile lab

You should get a json blob as a result of any ec2 instances running, or a response of “Reservations”:[] if nothing exists there.

Ansible and using the Security Token Service:

The AWS Security Token Service (STS) allows you to request temporary credentials to perform actions through the AWS API. We will need this to grab credentials for the lab account to create resources there.

We are going to create a basic playbook to create an EC2 Security Group that is capable of running directly or via STS credentials for a different account. The first thing we need to do is create a vars/default.yml file in our ansible directory, this will store some basic vars for this test playbook:

The next step is to create a vars/sts.yml. This will store vars needed to enable/disable the STS functionality, and the details needed to authenticate you. The values needed here are the same as what you will have placed in ~/.aws/config in the AWS CLI setup above.

Onto the playbook. It runs on the local machine where the playbook executes. And includes the 2 var files we just defined. Here’s a look before I describe it:

The first task uses sts_assume_role. This takes the default AWS credentials and variables, and requests temp credentials for the specified account/roles. Which it registers in variable assumed_role. The “when” command will only execute this task when the sts variable is true from the vars/sts.yml file.

The set_fact task takes the previously registered var, and retrieves the new credentials from the object. These values are then assigned to local vars for use in subsequent tasks and roles.

The last task is the actual creation of AWS resources. It could be any of the modules, but we will use ec2_group as an example. Here we take the newly set facts and use them to define aws_access_key, aws_secret_key and security_token. If we were to leave those lines out ansible would take our default credentials and attempt the resource creation with them. Likely that will result in authorisation, or missing resource problems. So we override the defaults with the new facts, forcing the task to connect to the API with the correct credentials. In this current form you would run the playbook like this:

$ ansible-playbook test.yml -e sts_mfatoken=123456 -vvvv

Which should result in the last couple of output lines looking like this, with a new security group in AWS:

PLAY RECAP *****************************************************
localhost : ok=3  changed=2  unreachable=0  failed=0

To MFA or not MFA:

Now MFA is always a good idea. And should be enabled on your IAM user account. But depending on your AWS accounts and trust configuration, you may or may not need MFA to assume roles. In my case MFA was required, so what you see in the examples above is what is needed to make the playbook work. This required the addition of the sts_mfa and sts_mfatoken vars. And the inclusion of “-e sts_mfatoken=123456” at the playbook runtime (with the number being the token from your mfa device). Otherwise you will end up with an authorisation error. If MFA is not a requirement, here are the modifications:

• remove mfa_serial from ~/.aws/config
• remove sts_mfa from vars/sts.yml
• run the ansible playbook command without “-e sts_mfatoken=123456”

Without STS?

This playbook was also designed to run directly without needing to assume a role. In vars/sts.yml just change sts: true to sts: false. This will cause any tasks with the when: sts clause to be skipped as they do not pass the conditional check.

But what about the credential lines in the ec2_group task? Well, where I specify the variable value, I have added | default(omit). This means if those variables are undefined, the module execution with omit those arguments. Forcing the execution to pull the default values (which are my default credentials).

Thats it for now……

Well that was probably a rough blog post. It was a bit rushed due to the things I was working on. But hopefully it is helpful. I hadn’t been able to find any real world examples of sts_assume_role in use. I will be dropping back to some more “Getting Started with Tools” in the future. Let me know if this helped you, or any improvements that could be made. Till next time, Automate all the things…