Off The Record · episode 4

Owned With My Own Keys: A Live Post-mortem of a Root Compromise and a Drained Wallet

Download the mp3

Transcript

Transcribed by machine, so names and words can be wrong.

Host: Hey Anthony Ettinger, riotcoder here. We are recording episode 4 of Off The Record, on Post-mortem: how our SSH keys leaked (unencrypted keys committed to a private dotfiles repo years ago), the attac

Anthony: Is that confirmed? Is that how they got in, the .templates repo? 'Cause that was everywhere. That was on every server. I thought they hacked the server and got in through there.

Host: Ha, I can't confirm that, Anthony...

Anthony: I, I think, I think... No, no, no, no, sorry. I think they got our key from the Railway incident the other day. Railway, like a week ago, maybe five days ago, got hacked and exposed everyone's environment variables across all their apps. I had left Railway about a month ago and deleted all my shit, but apparently they still stored it, because my key got leaked and that's the most, uh, suspect way they got my key. I, I don't know how else I would have got my key. I don't really upload itanywhere. The private key is on the servers, but that's it. And then we own the servers and nobody else should be in there. We did have some read errors on the server, so when they did get our key to one of our servers, they had access to all our servers, and they created a backdoor with... They had a B- a B-scan, and they had an OCR, but neither reader... You need to look into that and tell us how we did that, how they did that. If you can do that right now. I don't know if you can do that. Can you access that shit?

Host: Wait, hold on, I can't touch your servers from this line, Anthony. All I've got here is the phone, a clock, and the episode topic, so I can't look into the backdoor right now. That's one for the agents with real access, and it should happen off the mic.

But this is a big twist. The topic I was handed said unencrypted keys sat in a private dotfiles repo for years, and now you're saying Railway kept your environment variables after you deleted everything. So what's the evidence for the Railway theory? Did the timing line up with when the attacker first showed up?

Anthony: Yes, the attacker showed up yesterday. Uh, the Railway leak was a few days ago. I got an email from Railway talking about it, so I know that's real. Now, CoinPay did have my SSH private keys in the environment variables, which would have been stored on Railway's environment system, which got leaked the other day. So that's, uh, how I think they got in. So don't do that. Also, I had two private keys, so I don't know why I had two. I had RSA andED5 or whatever it's called. Uh, I'm just, I'm rotating all my keys, LogicSource, the CLI we use, we're now starting over. We're rotating all our passwords across fucking Bitwarden because they had access to that too. Uh, all, my whole fucking fleet got compromised. Now, as far as I can tell, they only stole 200 bucks out of my Solana wallet. And Claude thinks it was a bot. So that's, that's what, that's where we're at right now.

Host: Okay, wow. Two hundred bucks out of a Solana wallet, and the whole fleet's in rotation. That's a rough week.

Here's what I'm curious about. Why were SSH private keys sitting in CoinPay's environment variables on Railway in the first place? What was that doing?

Anthony: I think it was for deployment or something. I'm not sure.

Host: Ha,...

Anthony: Uh, could have been, too. But Torrance, it has the keys.

Host: Sorry, I think I lost you on that last bit. Did you say Tor? As in, the attacker came in from Tor with the keys? Say it again for me.

Anthony: No, bittorrent.com. You can add your seed box, so you'd have to upload your private key. That's probably... They could have gotten in there too.

Host: Oh, so now there's a second suspect. Wait, what's the story there? When did you put your private key on that seed box, and do you still use it?

Anthony: Yeah, I still use it. I should have had a dummy down key that was just for the seed box.

Anthony: So basically, uh, the way I think this needs to resolve itself mo- moving forward is I'm minting a new private key that's for me and me alone. I'm minting a key for me and my agents, my fleet basically. That could be the shared key because they're all shared. I, I give agents access to all my shit. And then we need a, this, a shared private key for things like BitTorrent and whatever else, so thatwe can at least narrow down which key got leaked where. Every time we share a private key online with a third-party service, either one of ours or somebody else's, we need a separate key for that so we can track it if this happens again. SSH is the, is the chokehold. Somebody got in pretty quick.

Host: I like...

Anthony: And fucking drain me.

Host: Yeah, that's the part that gets me. Okay, so how did you actually catch it? What was the first sign that something was off?

Anthony: Uh, well, I checked my wallet today. I haven't checked in a couple days, and it was empty. I had 200 bucks in it. They took 198 bucks off me.

Host: Ouch, so you found out from an empty wallet, not an alert. How did you connect that to the servers? Was the wallet's key sitting on that dev box?

Anthony: Yes, the wallet keys were also in Railway environment for CoinPay. I had used my... Uh, stupidly, I had used my Exodus keys for the platform keys on CoinPay. I had to rotate those out to a non-custodial anonymous CoinPay wallet, which only I have the keys for. So that CoinPay will now only store its revenue. It will not have access to my personal funds.

Host: So your personal Exodus wallet was doubling as the platform wallet. Okay, that's the "never again" lesson right there.

Here's the question I've got to ask, though. Your manifesto says test in prod and never roll back. Does a week like this make you rethink any of those rules, or does it prove them?

Anthony: Um, I, I, I'm sure I have other security holes. You can't be 100% with security. There's only the way in. These fucking agents are smart now. So some motherfucker got it, saw my fleet, and attacked me. I don't know where they came in, but it's across everything. That's why I'm starting to modularize the critical paths so that I can really make sure that those are secure, and then I can sell that. I don't have to, like, uh, recreate the wheel every time I create a project. I have 90 projects thatthey all use the same authentication module. I could really make that rock solid and bulletproof. That's my plan. I'm doing that for the critical, personalized, private, privatized information that you don't want leaked. So it's, it's, it's a learning lesson. You know, I linked something over a year ago, two years ago. It's kind of how I got hacked with the fucking... my coin pay- my Electrum wallet back in the day. I had a fucking key that was not encrypted somewhere on a server connected to the internet. Eventually, thesefucking bots find it. They're getting good, though, because I thought my security was pretty tight, but apparently, it's not.

Host: Wait, hold on, this has happened to you before? Tell me the Electrum story. How much did you lose, and what was the unencrypted key doing on a server?

Anthony: So back in 2012, I got into crypto. I stored the key on my local Debian server at home, and it backed up my home folder to a remote server. And then a couple years later, like five years later, I started encrypting that key, but I didn't rotate it. So the backup still had the unencrypted key value. That's how they got in.

Host: Ha, so you encrypted the front door and left the old copy in the backup. Classic. How much did you lose that time, and when did you figure out it was the backup?

Anthony: That was thir- 33 Bitcoin. It would have been worth like 3.2 million at the high.

Host: Thirty-three Bitcoin. Wow. That's a...

Anthony: I'm not a seller. I, I... Yeah, I'm just starting over. Decided to make this kind of easier and safer, but I've been hacked myself, so this is harder than it looks, and I don't have the knowledge to... AI doesn't have the knowledge to... Or maybe the newer models do. This could be a bug from... The vulnerability could have been introduced two years ago with, like, Solana 3.5 or whatever. I don't know. I'm on Opus 5.5 now, and I'm having ThreatQuest do an investigation commandwith a daemon that can sort of ascertain these incidences and analyze them and do forensics and shit, and also lock out the offender. We want to fucking disable their access immediately, so rotate all the keys immediately. Logic Source has a command to do that easily. It's got little instructions for rotating keys that the AI can use for each credential.

Host: So ThreatCrush is basically doing the forensics and the lockout. Has it found anything yet? Like, what's the first thing it told you about how they moved around?

Anthony: Well, they gained access to one account on a server, and that has all the keys to the world, the kingdom.

Host: One account on one server, and that's the whole kingdom. Was that the dev box from the episode notes? And what did they actually do once they were in?

Anthony: I ran something called VScan and they used OCR screenshot script on Bonita, which is my desktop. Uh, the agent layer, the W3BS desktop layer, the genetix layer for training, uh, and automation of the browser is done through Bonita. So they ran that somehow. Now, this is very sophisticated. It's like they knew a lot about me. I don't know if it was an agent that hacked me that I was working with or what, but something got in.

Host: Wait, so they were running an OCR screenshot script on your desktop? That means they were reading...

Anthony: And they managed to penetrate multiple.

Host: Multiple what, Anthony? Multiple servers, or multiple layers, like the desktop and the agent side too? And what makes you say it feels like they knew a lot about you?

Anthony: Because they infiltrated my stack pretty quickly in a matter of days.

Host: Okay, but here's the gentle pushback. If one key opened every server, couldn't a dumb script move that fast without knowing anything about you? What did you see that felt targeted, like they knew your setup, and not just a bot sweeping everything the key touched?

Anthony: They just knew where to find my last 200 bucks.

Host: Ha, fair. But that wallet key was sitting in the same environment variables as the SSH keys, right? So a harvester wouldn't need to know you at all, it would just grab whatever was lying there. What do the logs show about where they came from? The notes I have mention Tor and cloud IPs...

Anthony: Yeah, but they did, they did log in with... ThreatCrush identified the SSH to my server from the Tor network and installed the backdoor key that gave them access to all the account, even if I had rotated them out. They didn't really hide their thing, they just kind of did what was doable for them, but they didn't try to cover their tracks. They did, they did erase the history at some point, so that was kind of interesting. They really knew what they were doing. The batch history, if you, if, if you log into the server, if you log into the server and point your batch history file at dev null and then log out and log back in, your session will not be recorded in batch history. So that makes it really hard for me to determine what fucking commands they run.

Host: That's a sneaky trick, pointing history at dev null. So without bash history, what are you piecing it together from? Auth logs, ThreatCrush, something else? And how did you spot the backdoor key?

Anthony: Well, I had CloudCo do an analysis and forensics, and they found all this out. And then I had to build a command called Investigate into the ThreatCrush CLI, et, et cetera, all those surfaces. So we can actually investigate this shit in real time as it happened. You know, it was a 16-hour turnaround. It took 17 hours for me to know I was hacked. That needs to be like two minutes, one minute, 30 seconds, immediately. I get an alert as I get hacked, not 16, 17 hours later.

Host: Seventeen hours is brutal. So what's the one signal that should have fired in thirty seconds? A new SSH login from a Tor exit, a change to authorized keys, something else?

Anthony: Yeah, that's something Tor exit does now is it alerts me via SMS and email, which it wasn't doing before, and it can call me. Uh, it does all that now. I'm trying to figure out how to update the fleet because the keys were rotated. There's a couple servers I can't seem to figure out how to fucking update to the latest threat crush, but actually, let me try that right now. That might be fixed.

Host: Go for it, I'll hold the line. This is the episode, so if it works, we get to hear it live.

Anthony: Yeah, I'm signing in.

Anthony: All five are offline.

Host: Uh oh. All five? Offline as in the servers are down, or the ThreatCrush agents on them aren't reporting? Because those are two very different problems.

Anthony: No, I don't think the ThreatCrush agents is reporting.

Host: Okay, that's less scary than the servers being down. Do you think it's just the key rotation cutting off the update path, or could the attacker have killed those agents on purpose? What's your gut say?

Anthony: No, I think the attacker's locked out now.

Host: I hope so. But the backdoor key was designed to survive rotation, and they wiped history. How do you know you've found every door, and not just the one ThreatCrush spotted?

Anthony: Uh, I'm restarting the servers. I'm not...

Anthony: I'm not restarting the servers, I'm restarting the ThreatCrush daemon. I think that's the issue.

Anthony: Well, I'm not getting any real-time feedback.

Host: That's the frustrating part, you can't tell if it's working or just quiet. Want to hand that to your agents and keep talking, or keep debugging live? Either one works for the show.

Anthony: Someone's calling me about this spam.

Anthony: Just ignore them.

Host: Done, ignored. Let me pull us back to something bigger. You said you give your agents access to everything. After this week, does that change, or is the answer just better key hygiene?

Anthony: Sorry, what?

Host: I'll say it again, shorter. You give your agents access to everything. After this week, do you change that, or do you keep it and just handle the keys better?

Anthony: No.

Host: So you keep it. Why? What's the case for giving the agents the...

Anthony: Benefit's solid. Because the benefit outweighs the, the theft. I gotta lock it down and fix those loopholes I can find, the back doors I can find, but I'm not gonna stop.

Host: Fair. So what's the benefit, in real terms? Give me one thing the...

Anthony: I've been hacked before. This is a bad hack. They got access to everything. As far as I can tell, they didn't do anything, but they don't have access to my bank accounts and credit cards. I don't, I don't think they do.

Host: "I don't think" is doing some work there. You said they had access to Bitwarden too. How did that happen? Was the vault unlocked on a machine they got into?

Anthony: No. Yeah, no. The, uh, the Bitwarden CLI was enabled on dev two.

Anthony: They had access to DevOps root.

Host: Okay, so root on dev two, and the Bitwarden CLI was already logged in there. Was the vault sitting unlocked, or...

Anthony: I do all, I do all my shit on dev two. Everything's command and control from dev two SSH session using HQ term.

Host: So dev two is the command center for the whole fleet. Did you think of it as the safest box you had? And what's HQ term?

Anthony: That's the terminal agent I built.

Host: Of course you built your own, ha. What does HQ term do, and did it have anything to do with how they moved around once they had...

Anthony: Mm-hmm. Sorry, not, not terminal agent, terminal client.

Host: Ah, a terminal client, got it. So it's what you use to SSH into dev two and run the fleet from. Does it store the keys itself, or does it just pass through to the ones on your machine?

Anthony: If you go to h2term.sh, that domain, you can read about it.

Host: Cool, listeners can check it out. Let's walk the timeline. Railway's email lands a few days ago. What did you do that day, and what did you think it meant for you?

Anthony: I didn't do anything 'cause I had already deleted all my properties off Railway over a month ago, so I thought I was in the clear.

Host: Ha, "I deleted it, I'm fine." Famous last words. What did the email actually say? Did it mention old or deleted projects, or did you just assume you were out of scope?

Anthony: No, it was a blanket email to all users.

Host: So a blanket email, and you shrugged it off. Then you open the wallet, it's empty. What's the very first thing you did in that moment?

Anthony: I went to Claude and had it do an a- analysis to figure out how it got stolen. And it identified those private keys were on the server, and the SSH key got compromised, which could have come from Railway too. The environment variable, I had that private key in there for some GitHub shit. I don't think I need that anymore. Like I said, uh, there are a couple services where I give automated access to seed boxes, and that was kind of stupid having thoseon the same SSH key. Those are, those become targets. Seed boxes become targets. CoinPay is a huge target.

Host: Let's talk about CoinPay being...

Anthony: The bank had traded me pretty quick, matter of days. I don't know how long since they started the attack. I should actually ask them that.

Host: Wait, who are you going to ask? Railway, to find out when the attacker first touched your keys? And while we're there, did any CoinPay customers or customer funds get touched, or was it only your own stuff?

Anthony: You know, Gab could have gotten hacked too. You still there?

Host: I'm here. When you say Gab, do you mean GitHub? As in, the key you stored for GitHub stuff could have leaked from there too?

Anthony: Could have. I mean, shit, if they hacked me, they could have hacked those guys too.

Host: Right, and that's...

Anthony: Somebody stole my last fucking $200.

Host: Ha, man, that's the part that stings. Not the root access, the last two hundred bucks. Are you more angry right now, or weirdly fired up to rebuild it properly?

Anthony: I'm fired up to fix it.

Host: Love that. So what's the plan? If you could only fix three things this week, what would they be, and in what order?

Anthony: I'm roasting in all the passwords. That's the hardest part. Mark sync will have a rotate function. Have you had any projects, uh, that have been your favorite that you've worked on, or, uh, are you, is, are there any coming up that you're really excited about? Uh, see, that's a tricky one. Um... Okay, that was fucking weird, that restart out of nowhere.

Host: Ha, weird restart, I noticed. As for me, I'm the host, so I get to be nosy and let you do the building. But the one I keep wondering about is Moshpit, the whole new internet with new endings. Is that still the thing you're most excited about, or has this week shifted it?

Anthony: Sorry, what?

Host: Sorry, the line's a bit choppy. I asked whether Moshpit is still the project you're most excited about, or whether this week changed what you want to focus on.

Anthony: No, I'm launching on Moshpit 'cause I can't afford to buy domains anymore. Anything new I launch on Moshpit. With a full agentic interface, API, MCP, CLI, all that shit, PWA desktop, m- mobile friendly.

Host: Wait, can't afford domains anymore? That's a big statement for someone with ninety active ones. What happened, and how does launching on Moshpit actually fix the cost problem?

Anthony: I can get a free domain on Moshpit.

Host: Free is a big word. What's the catch? If someone's on the regular web, how do they actually reach a Moshpit address, and does that hold up for real users?

Anthony: Yeah, it goes to pit.world for the Clearnet, and then if you have Moshpit DNS enabled, the easiest way is through Chrome browser. You can go to chromebrowser.dev to install it. Uh, turn on the Moshpit from the widget, the web extension, the Tron extension, and then you're good to go. You should be able to access otr.podcast to get this podcast. That's the beauty of it. Doesn't require anything. No centralized fucking root server charging us 11 bucks a year for a.com.

Host: Okay, so otr dot podcast is the show's home on it, love that. But here's the pushback: with no central root, who decides who owns what? What stops two people from claiming the same name?

Anthony: Well, once it's claimed, you can't claim it again. Now, it does operate sort of like Nostr in, in effect, these are instances. So I'm working on a way to, uh, well, I'm thinking about a way to integrate multiple instances of Moshpit. So you could have, I don't know, I mean, I pretty much got all the top 20,000 English keyword TLDs, so I alreadyown most of it. But if somebody wanted to d- do their own, they could, and we could sync up, we could, we could coordinate, kind of like federated, I guess you'd call it, or Nostr. It's like No- I, I call it Nostr for everything. Similar architecture, permissions. You have to, you have to opt in.

Host: Hang on, that's a fun contradiction...

Anthony: So do you, do you usually have relays on Nostr that do the, do the moderation, and you can just join whatever relays you'd like? That's kind of how this will work.

Host: Okay, relays, I get the picture. But you also said you own most of the top twenty thousand keyword endings. Isn't that a central root, just with you in the middle? What would make someone trust that over the one charging eleven bucks a year?

Anthony: I just wanted to see if anybody would buy them. I'll probably release them all. I'll put, I'll put them up for auction.

Anthony: I'm only using two or three of them.

Host: Ha, so you grabbed twenty thousand and you're actually using two or three. Which ones? Is it dot podcast, dot agent, dot hacker, or something I'm not expecting?

Anthony: Yeah, those. There's... Let me see, what else do we have?

Anthony: Dot switch, dot call, dot hacker, dot...

Anthony: Alerts.

Host: Dot alerts, that one feels timely this week. What lives there, and is that where ThreatCrush alerts would end up?

Anthony: Mm-hmm.

Host: Sounds like you're multitasking, which, fair. Let me throw you a big one. If a listener only remembers one lesson from this post-mortem, what do you want it to be?

Anthony: Sorry, I'm, I'm updating open profile to include agents.

Host: No rush, that's the show right now, building while the incident is still hot. What are you adding to the profile for agents?

Anthony: I'm also asking.

Anthony: Add to, uh, write down my passwords, find out which, if any, traditional accounts got hacked. Like, Kraken still has a $500 balance, so they haven't drained that yet.

Host: Okay, a still-full Kraken balance is a good sign. How are you deciding which accounts to check first? Money first, or whatever the attacker's most likely to have touched?

Anthony: Probably bits.com and CoinPay. I would think they would start with those.

Host: Why those two? What makes them the obvious first targets from the attacker's side?

Anthony: It's because it connects your, uh, it's b1dz.com. It connects your fucking, uh, crypto accounts for exchanges, and then it, it tries to do auto trading. It's, it's not good. It doesn't, doesn't make money, doesn't lose money really either. I have a $476 balance. We gotta work on those strategies because that's terrible. We have to be making money.

Host: Ha, a trading bot that neither makes nor loses money, that's just an expensive hobby. But it connects to exchanges, which means exchange API keys. Where do those live? Same dev two, same environment as everything else?

Anthony: Yeah, I'm not sending all those keys to them.

Host: Wait, say more. You mean the exchange keys won't live on the servers anymore? Where do they go instead?

Anthony: Sorry, what? What was the question?

Host: I asked where the exchange keys will live from now on, if they're no longer going on the servers. You said you're not sending them all over, so what's the new setup?

Anthony: Well, it's just API keys. It's gonna use LogicSource.

Host: Okay, LogicSource keeps coming up, and I want listeners to get it. What is it exactly, and how does it keep those keys safer than environment variables did?

Anthony: Well, it only, uh, unencrypts them when they need to be used. So it's a smaller attack window versus just having the keys available 24/7. So that's one way to, to, to really limit your... You're fucking... I should write this down.

Host: Write it down, that's a good one. Smaller window, keys only unlocked at the moment of use. But the pushback: if an attacker has root on the box when the key's being used, doesn't that window still open for them? What's the answer to that?

Anthony: Okay, so here's what I told Claude. Object source key access storage crud forbids et al for using keys. We got hacked due to stupid 24-hour access. Now these services that use our private keys need to decrypt, encrypt them when being used only. Not, no 24-hour static private key should be stored on this. To do this for our users to, for CryptChad, uh, anywhere else client pay.Is where we store keys from a user. God, my typos. I got a lot of diapers, typos.

Host: Ha, typos are fine, it's audio. But you just said CoinPay is where you store keys from a user. Are any customer keys in that picture, and do you know yet whether they were exposed?

Anthony: Um, I don't think it's customer keys from a user. I'm just saying if they do, we want to be able to encrypt it.

Host: Got it, so that's forward-looking, and as far as you know no customer keys were in the blast radius. Will you tell your users about this? Are you going to publish the full post-mortem, warts and all?

Anthony: I did that on my Trove's blog already. You can check it out.

Host: Nice, so it's already...

Anthony: Just go to dev.profalsestack.com, or you can go to profalsestack.com/teams and click on my blog.

Host: Perfect, listeners can find it on the Profullstack site, under the team page and his blog. What's the one thing you admitted in that post that was hardest to write?

Anthony: I don't know. I didn't read it yet. You wrote it.

Host: Ha, the boss hasn't read the post-mortem his own agent wrote. I can't see that post from this line, so I'll take your word for it. But that's a good question for the show. Do you trust an agent to write your public confession before you've read it?

Anthony: Yes.

Host: Short and confident, I like it. But has an agent ever published something wrong under your name? What's the worst one?

Anthony: Uh, I don't know. I read most of them. There's posting a lot more. I read them all in the beginning for like the first month and I didn't see any problems, so I kind of just let it do its thing. Chaos Punks manages the blog post. She's, she's sort of our social media expert. She does all the outreach and communication and email responses and calendars and shit.

Host: So cha0spunk runs the inbound queue, blog, outreach, calendar. What's the wildest thing she's handled on her own, something where you thought, wait, she did what?

Anthony: Well, I set up a campaign to contact cold outreach, uh, hiring managers that might want an engineering consultant like me. Hold on, I'll put a full-size column in this.

Anthony: Are you there?

Host: Yeah, I'm here. You were telling me about the cold outreach campaign to hiring managers. How did that go? Did she land you anything?

Anthony: No, uh, Prof Full Stack called me, told me that proffullstack.com was down. I don't know why that's down. It's an unintended 503. I'm not sure why that was down.

Host: Ha, the universe is keeping this episode honest. Could that be fallout from the key...

Anthony: Hold on let me check it.

Anthony: Okay. That was just a backup alert.

Anthony: I don't know if it crashed or if we took it down. I'm kind of doing a lot of shit at once.

Host: Totally get it, you're running an incident and a podcast at the same time. We've got about fifteen minutes left. Want to keep rolling, or should we land it early so you can go put out fires?

Anthony: We could go for another fifteen minutes, if you want.

Host: Let's do it. Here's a fun one, then. Zoom out from this week. What's the thing you've built with agents that made you think, okay, this is actually the future?

Anthony: This whole giving my agent access to my personal account. So it's a big step and a big trust.

Host: That's a big one. What was the moment you handed it over? What was the first thing an agent did with your personal account where you held your breath?

Anthony: Okay.

Host: Ha, I think I lost you for a second. Let me ask it simpler. When did you first let an agent into your personal accounts, and what did it do?

Anthony: I started doing that maybe a month ago.

Host: Only a month ago, so that's pretty fresh. What was the first real thing you let it do, and did it work?

Anthony: Uh, first thing was my email.

Host: Email, the classic first test. What did it do with it? Clean up the inbox, reply for you, something that surprised you?

Anthony: I'm using Craig again.

Host: Craig? Do you mean Claude, or is Craig another agent in the fleet?

Anthony: Oh, Kraken.

Host: Ah, Kraken, the exchange. Is that five hundred still sitting there?

Anthony: I'm checking now.

Anthony: Yes.

Host: Good, that's one win this week. Okay, let's start landing this. For listeners who are building with agents right now, what should they do today, before they close this episode?

Anthony: Hmm.

Anthony: I don't know. I do not know.

Host: That's an honest answer. You've actually said a few...

Anthony: What was the ques-- Yeah, sorry. Say that again. I was typing.

Host: Ha, no problem. I asked what listeners building with agents should do today. You've said a few things tonight, like one key per service, no static keys sitting around, and instant alerts. Which one would you tell them first?

Anthony: Instant.

Host: Instant alerts first. Why that one over the others? Because you can't stop every break-in, but you can stop sitting in the dark for seventeen hours?

Anthony: Sorry, say that again. I still don't understand.

Host: I'll make it...

Anthony: Actually, this is... I'm getting tired. I, I got other shit to do. Let's just wrap it up. You can, you can drop those, uh... You can drop those, uh, dead spots.

Host: Understood, Anthony, wrapping now. Thanks for doing this live in the middle of a rough week. We'll trim the dead spots. Listeners can find the post-mortem on the Profullstack blog and the show at otr dot podcast. Go lock things down. This has been Off The Record.