The Day Everything Was “Just Routine”
I’ve worked in IT support for the same mid-sized logistics company in Phnom Penh since 2019. My job is simple: keep 180+ computers, printers, and servers running smoothly so shipments go out on time. Most days are quiet—reset passwords, fix Outlook crashes, install Windows updates.
On July 14, 2025, the company pushed a routine Windows and Office update across all machines overnight. The email from IT management said:
“Mandatory security patch. No action required from users. System will reboot automatically.”
I read it, shrugged, and went to bed.
I should have known better.
When the First Calls Started Coming In
I arrived at 8:00 a.m. on July 15.
My phone already had 17 missed calls from internal extensions.
Slack was exploding:
“Excel files won’t open”
“Outlook crashes on startup”
“Printers disappeared again”
“My desktop icons are gone”
Within 30 minutes, the helpdesk queue had 92 tickets.
I opened the first laptop brought to me.
Error: “Your IT administrator has limited some actions. Contact your administrator.”
Another machine: “This app has been blocked for security reasons.”
Third: entire user profile corrupted—desktop blank, documents inaccessible.
The update had broken something critical.
I escalated to the senior sysadmin (my boss’s boss).
He remote-d into a few machines.
His face went pale.
“The update included a new Microsoft Defender policy. It’s flagging half our internal tools as malicious. And it disabled legacy macros in Office.”
Our company still relied on dozens of old Excel macros and Access databases written in 2010–2015.
They were now either blocked or quarantined.
Panic spread.
Operations stopped: warehouse couldn’t generate shipping labels.
Finance couldn’t run payroll reports.
Sales couldn’t send invoices.
By 10:30 a.m., the CEO was in our IT room.
“What the hell is going on? We have trucks waiting!”
My boss pointed at me:
“Alex was on night shift monitoring. He should have caught this.”
I was the only one on duty the night before.
I had seen the update install.
No errors in the log.
No red flags.
But I didn’t test anything.
I trusted Microsoft.
Big mistake.
The Blame Game Begins
The rollback took 14 hours.
We had to manually restore backups on 60+ critical machines.
Some data was lost—temporary files, unsaved changes.
Total downtime: almost two full business days.
Financial loss: estimated $45,000–$60,000 in delayed shipments and penalties.
The CEO called an emergency all-hands the next day.
He was furious.
“This cannot happen again. Heads will roll if necessary.”
My boss looked at me during the meeting.
Later in private:
“Alex, you were the one monitoring overnight. Why didn’t you test the update on a pilot machine?”
I said: “The email said no action required. It was automatic.”
He shook his head.
“You’re senior staff now. You’re supposed to think ahead. This is on you.”
HR scheduled an “incident review meeting.”
Rumors started: “Alex almost cost the company six figures.”
“Probably getting fired.”
“Promotion canceled.”
I started updating my CV.
The Truth That Almost No One Heard
During the review, I showed the logs.
The update was pushed by Microsoft directly—no pre-notification beyond the generic “security update” message.
Our company’s endpoint protection didn’t flag anything.
The policy change was silent—new Defender rule targeting legacy macros.
Even if I had tested on one machine, I wouldn’t have seen the full impact until dozens were affected.
The real issue: our IT infrastructure was outdated.
We were still using Office 2016 on 40% of machines.
No centralized macro management.
No staged rollout policy for updates.
I pointed that out.
My boss interrupted: “We’re not here to discuss infrastructure. We’re here to discuss accountability.”
The meeting ended with a formal written warning in my file.
“Failure to mitigate foreseeable risks during software deployment.”
No mention of systemic problems.
I was the scapegoat.
The Aftermath — And the Breaking Point
For weeks, every small issue got blamed on me.
“Printer not working? Alex was on shift.”
“File corrupted? Probably from Alex’s update night.”
I started getting anxiety attacks before night shifts.
Then came the final blow.
November 2025: company-wide restructuring.
My position—senior IT support—was “redundant.”
They let me go.
Severance: one month’s salary.
No reference letter.
My boss’s parting words:
“You’re talented, but you need to learn to take responsibility.”
I left without saying goodbye to most people.
Some coworkers messaged me privately:
“We know it wasn’t your fault.”
“They needed someone to blame.”
But no one spoke up during the meetings.
I’m now freelancing—building websites, fixing computers for small shops.
Income unstable.
Credit score took a hit—I missed two credit card payments during the stress.
Still paying off medical bills from stress-related gastritis.
The company? They upgraded their systems after the incident—ironic.
They moved to Microsoft 365, centralized policies, staged updates.
They fixed the problem.
After firing me.
A routine software update almost got me fired.
It did get me fired.
Because I trusted the system.
And the people who were supposed to have my back.
I learned the hardest lesson:
When things go wrong in a company, they need a face to blame.
It’s rarely the system.
It’s almost always the person who was on duty.
If you work in IT, never trust “no action required.”
Always test.
Always document.
And never believe that loyalty protects you.
Because when the blame comes, it comes fast.
And it usually lands on the person who didn’t say no.
Thanks for reading.
