SARCASM Alert!
This post is intended to be a mixture of sarcasm, different types and shades of humour, and actually is fundamentally entirely for educational purposes.
Please, please, take it as such.
It is not intended to offend or upset, though I do understand that due to the linkage to terrorism, it could be seen that way.
That aside, we have to deal with this in IT a lot, so let’s get on with the content and hopefully you’ll learn a thing or two on the way.
If you have comments on this post you can give them directly to me privately or call me out publicly, I don’t mind either way. I will have a GitHub Issue for comments which you will be able to see at the bottom.
There are errors when it comes to spelling, which are purposely left in, especially due to me disagreeing with some of them. Please don’t comment on those other than if it makes reading more difficult and inaccessible.
TL:DR
ITOps is hard, but doesn’t need to be, and we are in a much better place than we were 10, 5, 3 or even a year ago.
Providing the right permissions across your IT Estate to enable development velocity, whilst ensuring ideal cost effective measures are in place.
Sometimes we make mistakes like rm -rf / are well known, and tooling like terraform, as well as others, when not controlled properly, can cause all sorts of headaches for you as an Individual or as part of any organisation that you may be a part of.
What is ITOps Terrorism?
Broadly speaking, ITOps Terrorism, if you decide to use that term, can include the process where someone invokes a series of destructive tasks, whether purposely or accidentally. It can also be where one leaves a backdoor into an environment.
When done purposefully, by a trusted actor, it is done under the Chaos Engineering mindset & is a trusted and tested way of stress testing your IT Infrastructure and Application resiliency.
Examples of this include 3rd party penetration tests as well as running internal war games. This should not be confused with Games like Age of Empires, Total Annihilation, Command & Conquer or Supreme Commander, to name a few. As an avid gamer, having played most of the games in those series, I do wish we would move away from having actual wars, and settle our differences using virtual, non-destructive warfare like via these and other games. I can continue to dream on that may happen in future.
When done accidentally, by a trusted actor, it causes stress, application outages, which in turn, cost both time and money & can lead to the loss of jobs. We know the rm -rf / and deleting C:\Windows\System32 but that’s as they say “just the tip of the iceberg” and there are many other similar actions like this too.
An example of this is your developers having Prod and Dev access and running a destroy action, on the wrong environment. Something I did to my production environment many years ago. Whoopsie - lesson learn’d the HARD WAY!
When done purposefully, by an untrusted/unauthorised actor, this can often be termed as Corporate Espionage and can have devastating impact to the business as well as any downstream organisations that are reliant on that business, including other IT vendors, as well as countless other downstream organisations and individuals.
An Example of untrusted purposeful ITOps Terrorism, is ransomware, or Command & Control black hat hacking, to at a later day extort you, your org, or any other person or organisation connected to you.
Chaos Engineering / Theory
Chaos Enginerring (as opposed to Engineering) is fundamentally the ITOps implementation of what Chaos Theory teaches us to be aware of. It helps us Err on the side of caution, by implementing guidelines, processes and importantly guardrails where we can.
Simply put, in my own words & not those of any AI Chat tool nor what you can read on Wikipedia or other places, is that no matter how well you design for stability, whether in software development or in general life, you can’t truly feel the benefits it brings, if you haven’t, during a previous point of your life, had to equate for irratic, chaotic and utterly frustrating peroids along the way. You don’t get the level of resilience & determination required without being dragged through chaos & suffering of one form or another, no matter how extreme those periods had been.
This is one reason why we stress test systems, using repeatable tests, as to feel good about our applications and them working when a component has a hiccup or “throws a paddy”. We often do this with people too btw.
We also should be designing and using destructive testing practices, where we induce some pre-determined chaos, as to mimic “thowing a paddy” as this allows the testing of our chaotic response process/es. Regular “fire drills” are important to carry out. As are the irregular, unscheduled ones too. This is a fundamental part of what Business Continuity Management is all about, something many businesses could do better in.
What is ITOps Counter-Terrorism?
ITOps Counter-Terrorism is where we declare the state where we need to have the highest level of response as well as the quickest route to recovery. This is in ITIL called a Sev1 Major Incident. You could call it Defcon 1, or use the UK JTAC’s CRITICAL or any other naming convention you choose.
We would call this under ITIL this is called Major Incident Handling, but we could even go as far as to take a leaf out of both military and policing and invoke a similar process to their Gold Command, Counter-terrorism processes, or frontline responses. In Medicine, we would invoke a CODE BLUE and then invoke recovery via ICU & then all the way back to community care.
The process for this is relatively similar no matter how prepared your organisation is for this as well as lower states of emergency.
If you truly have a Defcon 1 level incident, you likely need to initiate a full organisation wide shutdown, slowly building back up until you get back to your intended organisational desired operational state. This could be months or you could have to perform emergency organisational surgery and amputate a part of your business. Ideally this doesn’t have to happen, but if so then this is absolute Wurst of all the possible sausages you could & should be enjoying as part of your evening meal, or your breakfast.
Good Operational Practices
There are so many other things you must be aware of when it comes to planning for the worst or even worst trying to recover when you don’t have a plan for recovery, as such the following is a starter of what you should think about.
Backup/Restore
What of your data, do you have full visibility of how you reliably backup and restore without being held to a single point of failure?
If you don’t know this, spend a day or two starting to understand this. Especially if you are using SaaS products as these can be teriffyingly terrible (sorry for such a bad pun) and can hold you to ransom, with your data being a hostage. That is also the same case if you end up getting hit with Ransomware, which falls under ITOps Terrorism.
Backups need to be done in a way that can be easily be restored, and as any IT Pro & especially your Data Engineers, will tell you, a backup is useless if you’ve not tested the restoration of it.
Identity
Identity is a critical area, which is why organisations are always coming under attack, with a view to gain the keys to your Identity service of choice.
Take this one bit of free advice, Multi-factor Authentication is a 100% requirement, making use of strong verification methods, that are not, I repeat, NOT, and I mean not now not ever, tied to a mobile phone number with SMS based Time Based One Time Passcodes. 2025 needs to be the year where these are a fall back, or removed as a method for MFA attestation that you have control and access to the accounts you must be have secured and require access to. It also needs to be the year were service providers do better to support and not cut off customers in difficulty as a phone number is a verifiable data point, especially if it is long lasted.
Good Password Manager Practice
Whilst NCSC and other security guidelines recommend using passphrases for your password manager, (which I don’t disagree with) it is a good security hygiene practice to have a number of regular and irregular account checks as well as a rotation of any passwords or passphrases.
If you have a Password Manager, I’d recommend the following approach & if you don’t go and get one that isn’t built into the browser!
-
Rotate your Master Password at least once every 5 Years
- Ideally this should be more frequent (like once a year) but not so frequent that it becomes cumbersome and annoying across devices.
-
Check for leaked passwords of accounts in your Password Manager
- Rotate them ASAP if leaked.
- Rotate them even if unique on an irregular schedule
-
Check for accounts you just do not and will not use again and close them & request the deletion of your data.
-
Stop creating accounts across services if you don’t need them.
- Less, is more, as they say.
-
Have a backup device, which you regularly update and log in to your critical services, ideally not using their App if it has a website.
- Whilst not great and another expense, an old cheap laptop or tablet, or older phones make good backup devices
- But beware if they are not getting software support as that makes them vulnerable to attack - so minimise their network access where possible.
-
Change up the services providing your core identity products every so often
- Change DNS, Email & Mobile Phone numbers that you use for critical services here and there.
- Misspellings is a great way to do this (although often seen as negative)
- Aliases (often confused with alternative identities) are incredibly useful.
- For example I use aliases across a number of different services.
- I use multiple different “Identities” in my lab environments.
- The difference between the two should be aspects of traceability, unless you actively decided to leave that trail.
-
Segment critical account logins from everyday low value accounts
- Low value accounts are all those forums or job boards or other things that you needed to have a login to.
- Do not use social accounts to log in to other services - set aside unique email addresses where possible
- MFA EVERYTHING - see 2fa.directory for services that are good and have MFA enabled.
-
Optionally migrate from one password manager to another
- Or segment by using multiple for different purposes
Identity Sum Up
Identity is as critical as the door you have to your home and the locks you have installed on it. Sure you can blow the door up or smash through it with a tank, or bypass it and jump in through the open window but for most of us we love the illusion of the safety of our own homes.
This is perhaps, because we often don’t expect trouble at our door, and as such, expect that we can “fly under the radar” and not be attacked. This could not be further from the truth.
I don’t want to alarm anyone, but we are in an international state of extremely heightened global tension. We will see more and more people and businesses targeted in cyber espionage practices whether this be by attacker groups backed by nation state actors, ransomware groups, or by individual script kiddies, often powered by Coffee, Red Bull and abusing AI Chatbots to help them in building l33t hax0r skills to show off to their friends at school. Whether they don’t understand, or don’t care, or both, about what they are getting into with these, it often is for the funzies or in the worst case, because they want the easy path to the lifestyle of of many on social media these days.
I have 18 years of good solid sensible research behind me in this area alone. I haven’t published, let alone discussed, at least not in the open, a large amount in this area. However I will so in future.
Whilst I can’t discuss some things, I do know that the Identity Team at Microsoft (along with so many others too) did take on board a number of my comments over the years.
Which reminds me, I need to play with Active Directory and Windows Server 2025, along with a other identity products (and other products in general) in 2025.
Deployment
This is one of my favourite areas of technology. Defining the configuration and then going and deploying the technology we use in our day to day lives.
Terraform is a valid choice, however, when push comes to shove, it would not be mine & as such were possible I choose to avoid it. There are people out there that like/love it, much like I do with the experience I have in scripting and deploying using a mixture of PowerShell & PowerShell DSC, as well as ARM Templates or Bicep for Azure.
I try to minimise code complexity and the understanding I have gathered over the years allowed me to build tools like the ARMTemplateComplexity PowerShell Module which was a fun project that I need to spend more time with in the future, even if Bicep is more favoured now. I may update that for bicep. I might not. 🤷♂️
Automated deployments
Whatever tool we use to deploy things we should make as much of it automated as possible.
Terraform, or OpenTofu or another tool, is just that, a tool. Use of them, is better than not using them.
The below is just a segment of what I enjoy using.
PowerShell
At time of authoring this post, I am in multiple different working groups for PowerShell, Interactive User Experience, Cmdlets & Modules as well as the Engine group. I want to thank all that I have interacted with across these & other groups, especially after losing my dad to cancer in 2022 and then being made homeless in 2023.
This technology, along with the community behind it, actually is one of the things over the years that has kept me going when I have struggled the most. The same applies to a number of other communities too, like those in the M365 & Azure spaces & others outside of tech too.
PowerShell, which is the scripting language I choose to use daily, is for me, fun & enjoyable to write in.
Others absolutely hate it, but we have a really good community around it. One that, whilst has changes in how active some are and how engaged they may be, does keep getting stronger and stronger each year.
DSC
This brings me nicely on to using DSC (Desired State Configuration) which I could happily write or talk about for days at a time, I’m not the only one either.
This was a PowerShell only technology, or should I say, a PowerShell backed technology.
DSC v3, is currently being built by mostly Steve Lee (Head of the PowerShell Team) and is looking to be a great new direction for DSC.
I still like DSC v1.1 but I am expecting once I get more ability to spend time with v3 that this will change very quickly, especially with more management of cross platform resources going forward.
Where’s the Terraform link?
Terraform has the terraform destroy command, it is has also been linked with terrifying when Microsoft released Azure Terrafy, & then renamed it to aztfexport, which may have been a sensible decision.
It is purely coincidental, that we can connect these things together in this way.
But terraform and it’s history, especially it’s recent history with the licensing change as well as the split in community to OpenTofu, has been interesting to watch, as an outsider. Much like many other, very not funny things, happening across the world as of late.
I hope for us to see some calmer times ahead in 2025 and beyond.
Sum Up
The world is changing, and we are all on this bus together. We must get better at IT Operational Management, whilst enabling developers to do what they can do best, building new things whilst fixing or enhancing all the things we have out there. I enjoy being “on the fence” of the IT Pro & Developer & seeing how we can bring lots of thi
I may have been unkind in picking on terraform in my title to this post, thought it’s not everyday you notice an opportunity to try and bring some humour to hard to discuss topics whilst being educational at the same time.
But I will say, well done to the team & community behind terraform for releasing v1.10.4 just a few days ago. I know this is yet another great milestone and whilst it’s not my tool of choice, I am glad it exists for those that not only choose to use it, but enjoy doing so. No matter what differences in opinions we may have, the single most important thing when it comes to working in this industry is enjoying what we are doing, especially if it’s having a good impact and makes you feel good about your achievements, because I know that I am happy about my previous achievements, and am looking ahead with a positive mindset for not only my future, but that of us all.
As you can likely tell, I can happily type and type and talk and talk & this will be one of the areas in 2025 and beyond you will see more from me. More on that soon!
How can I help?
Well done on getting this far, its been a long road in his post alone, which I how I like it.
If you do feel like reading some more then please do go read my recent update to Announcing Re-Initialise and get in contact with me. It would be great if I could start getting some paid work or help towards crippling debts, replacing my aging IT equipment and so much more.
I am UK based but can work around any timezone easily enough, thanks to an irregular sleep cycle. Right now I need to be mostly remote, because I have ongoing physical health issues, that make travel, more stressful than it needs to be. It’s manageable, but does make things more challenging than I’d like, one of the joys of getting older after a number of stressful periods in life.
Music behind this post
As usual I listen to lots of music whilst working on this post & as part of my way to continue to share more where I can, I will be adding to all future posts a section like this.
Having the ability to have my music loud is a big part of why I really enjoy remote work vs working in offices, though even when I have worked in offices it has never stopped me having a solid listening history, which I will write about another time.
I may add links to these below & publish them in a playlist another time. However I am thinking on how I best manage this going forward, especially as they may not be in a single streaming service. This most likely will via a database of sorts. I know if I had Azure credits to play around with I would look at building an Azure Function & Azure SQL for this. (heck I may even just go with a json file published to a gist)
The opening theme for Friends Megadeath - Addicted to Chaos Faithless - Mass Destruction Camo & Crooked - Turn Up (The Music) Basement Jaxx - Red Alert Nu:Tone - Bleeper High Contrast - Eternal Optimist Machine Head - Through the Ashes of Empires (album) Gloria Gaynor - I Will Survive Machine Head - Imaginal Cells Above and Beyond - I am What I am Jack Johnson - Better Together Metallica - Frantic Nero - Two Minds Super 8 & Tab - Burn Slipknot - Snuff Lloyd - Tru H “Two” O, Platnum - What’s It Gonna be? Red Hot Chilli Peppers - Dark Necessities Morgan Page, Lissie, deadmau5 - The Longest Road (feat Lissie) - deadmau5 remix London Elektricity - The Plan That Cannot fail Beautiful Lies - VIP - B-complex & plenty more
I could post everything I’ve listened to whilst building this however that’s a brief insight to the diverseness of my music taste for now.
The lyrics to Tru by Lloyd, So please accept me for who I am, And please accept me for what I do is so fitting, as is I just wanna talk to you and tell you how I feel in What’s it Gonna be?