I use AI coding tools daily on paid client work — not as an experiment, as the way the work gets done. So this is not a post about how AI-generated software is junk. It isn’t, and I would be arguing against my own desk.
What I am arguing is that these tools shift where the risk sits, and the shift is predictable enough to check for.
But the failure mode is consistent. AI is very good at the part of a project you can see — screens, forms, reports, the demo that made you write the check. It is much less likely to produce the part nobody asks for out loud: backups, deploy scripts, error alerting, server-side access control, a README that works. Nobody types “and set up a tested restore procedure” into a prompt, so it usually isn’t there.
Predictable gaps are checkable gaps. Here are ten a non-technical owner can verify — by asking one question, or by clicking around. For each I’ve written what a good answer sounds like and what a bad one sounds like, because bad answers are almost never “no.” They’re a confident sentence that changes the subject.
1. Can you open the source code right now, in an account you control?
What to check: Log in to GitHub, GitLab, or Bitbucket yourself. Not a screenshare of your developer’s screen — your login, your company’s organization, your name on the owner row.
Good answer: “Here’s the invite to your organization. You’re the owner, I’m a collaborator you can remove.”
Bad answer: “It’s in my repo, I’ll send you a zip when we’re done.” Or: “It lives in the builder tool, I can export it.”
Why it matters: A zip is a snapshot, not a project. The change history — who changed what and why — is most of what lets the next developer work safely, and a zip throws it away. If the code exists only inside a builder tool under your developer’s login, you don’t own an asset. You own their subscription.
2. Are the cloud, domain, and vendor accounts in your name, on your card?
What to check: The billing email and payment method on every account the app touches — hosting, domain registrar, database, email sending, payment processor, any paid API.
Good answer: A one-page list of every account, root credentials in your company’s password manager, your card on file, and your developer holding a separate limited login.
Bad answer: “Don’t worry about it, it’s on my account and I bill you back.” Or: “I’d have to check which email I used.”
Why it matters: This is the most expensive item here to fix later. A domain or cloud root account under someone else’s personal email is a hostage situation you won’t notice until you need it — a rate change, a lapsed card, a relationship gone quiet. Recovering an account you were never named on means a support queue and weeks of waiting.
3. Ask them to run the test suite while you watch
What to check: You don’t need to understand the output. Watch for three things: a suite exists, it runs, everything passes. Then ask whether it runs automatically before anything goes live.
Good answer: One command, some scrolling, a summary line saying everything passed — and yes, it runs on every change before deploy.
Bad answer: “A few of those are out of date, I’d have to fix them first.” Or: “I test manually as I go.”
Why it matters: Writing tests is something AI genuinely does well, so their absence is telling. What it doesn’t do is keep a suite honest as the code changes underneath it — and a suite that has been failing for two months is worse than none, because everyone has learned to ignore it. “I test manually” means the only thing between a small change and a broken checkout page is one person’s memory of how the system used to behave.
4. “Are there any passwords or API keys stored in the code?”
What to check: Ask it directly, then ask the follow-up that does the real work: if we had to change every password and key this app uses, do we have a list, and how long would it take?
Good answer: “None in the code — they’re in environment variables or a secrets manager. Here’s the inventory. Rotating all of them is about an hour.”
Bad answer: “They’re in the config file, but the repository is private.” Or a pause, then “let me look.”
Why it matters: “The repository is private” isn’t a security control, it’s a bet — that it’s never made public by accident, that no laptop holding a copy is stolen, that no contractor keeps a clone. Committed credentials also survive in the change history forever; deleting the line later doesn’t remove it. Even a shrug is useful here: it means nobody has a credential inventory, and you need one regardless.
5. “When did we last restore from a backup?”
What to check: Note the wording. Not “do we have backups” — everyone says yes to that. Ask when someone last took a backup file and rebuilt a working copy of the system from it.
Good answer: “Nightly, stored somewhere other than the server itself, and I last restored one into a scratch environment on this date.”
Bad answer: “The host handles backups.” Or: “They’re on the server.”
Why it matters: A backup you have never restored is a hypothesis. They fail in quiet, boring ways — the job stops running, the disk fills, the export captures the database but not the uploaded files. And a backup stored on the same server as the app protects you from a mistake, not from losing the server. A backup is theory until you’ve restored from it.
6. “If you were unreachable for a month, how would we ship a fix?”
What to check: The answer should describe a written procedure, not a person. Follow-up: has anyone other than you ever done it?
Good answer: “It’s documented — push to the main branch and the server updates itself,” or “run this one script.” And someone else has followed those steps at least once.
Bad answer: “I’d log into the server and update the files.” Or: “It’s a bit involved, there are manual steps I’d have to walk someone through.”
Why it matters: If deploying lives in one person’s head and fingers, you don’t have a maintainable system — you have a retainer you can’t cancel. Manual deploys are also how sites end up half-updated at 4pm on a Friday. On my client work, releases go out through a webhook: push to the main branch, the server verifies the request and updates itself. The particular setup doesn’t matter. That it’s written down and repeatable by someone who didn’t write it does.
7. “How will I find out the app is broken?”
What to check: Ask what gets notified when something fails, and ask to see the alert history from the last few months.
Good answer: “Error monitoring is running — Sentry or similar — and it messages me when the app throws an error. An uptime checker hits the site every minute. Here’s the history.”
Bad answer: “Customers will let you know.” Or: “There’s a log file on the server.”
Why it matters: A log file is evidence, not an alarm. Without monitoring, the clock on an outage starts when a customer gets annoyed enough to call — the slowest and most expensive detection method there is. AI-built systems fail silently more often than most, because generated error handling tends to catch a problem and continue rather than report it. The app looks fine. The nightly import just quietly stopped three weeks ago.
8. The locked-door test — run this one yourself
What to check: Copy a URL from somewhere sensitive: an admin page, a customer record, an invoice. Log out, open a private browsing window, paste it in. Then log in as your lowest-permission user and try again. Then change a number in the URL and see whose record loads.
Good answer: Every attempt bounces you to a login screen or a clean “you don’t have access” page.
Bad answer: The page loads. Or it half-loads — the layout appears, then the real data fills in a second later.
Why it matters: This is the difference between hiding a button and locking a door. It’s a common gap in quickly-built software: the app gets assembled screen by screen, and the permission check that belongs on the server gets written in the browser instead, where anyone can walk around it. If you find one, fix it before anything else on this list. It’s the only item here that can be exposing customer data right now.
9. The cold-start test: could a new developer run it tomorrow?
What to check: Ask for the README, then ask how long it would take another developer, with no access to you, to get it running. Better still, pay a second developer for one hour to actually try.
Good answer: A setup document listing prerequisites, the commands, how to load sample data, and how to run the tests — and someone other than the author has followed it end to end.
Bad answer: The generic README the framework generates. Or an impressive-looking document describing folders and features that don’t quite match the code.
Why it matters: AI writes documentation eagerly and confidently, including documentation for code that changed the following week. Confident, wrong docs cost the next developer more time than no docs at all. That paid hour is the cheapest due diligence on this list, and the closest thing to a real answer to “what happens if my developer disappears?”
10. “What’s the plan for updates and security patches?”
What to check: Ask what the app is built on, when its components were last updated, and whether anything it runs on has passed its end-of-support date.
Good answer: “Versions are pinned, there’s a scheduled update pass, automated pull requests flag anything with a known vulnerability, and the SSL certificate renews itself.”
Bad answer: “It works, so we don’t touch it.” Or: “Updating might break something.”
Why it matters: Software rots in place even when nobody edits it. The version of PHP, Node, or Python underneath your app has a support end date, and once it passes, security fixes stop arriving. Certificates expire. Generated projects also lean toward whatever versions were common in the model’s training data, and pull in more third-party packages than a person would. “Updating might break something” is usually true — which is exactly why you want item 3. Those two are the same problem.
The disappearing-developer test
If you do nothing else, write down the answers to items 1, 2, 6, and 9 and keep them with your operating agreement. Repository, accounts, deploy procedure, cold start. Those four decide whether your software is an asset you own or an arrangement you’re inside of. The other six decide whether it’s still working a year from now.
This is not an argument against AI-assisted development
I’d rather have a system built fast with AI and then hardened than one built slowly without it. The speed is real. What changed is the ratio: when features are cheap to produce and operational groundwork is not, projects drift toward all front and no foundation — and that imbalance never shows up in a demo. It shows up the first time something breaks.
Worth saying plainly, too: most of what’s on this list isn’t a developer being careless. It’s work that was never in scope, because you didn’t know to ask and they didn’t want to pad the quote. That’s a fixable conversation, and often the same developer is the right person to do the fixing. They just need the list.
If you’d rather someone else ran the checklist
That’s the kind of work I take on, for owners in exactly this position — you paid for software, it mostly works, and you want to know what you actually own before spending more on it. Pricing is hourly with a clear estimate up front; longer relationships move to a retainer.
Going through the list together covers the same ground it does above: what you own and what you don’t, what’s exposed, and what it would take to close each gap. Whatever it turns up, you can hand to your current developer — it doesn’t have to be me who fixes it.
What it turns up is usually unglamorous and specific. On one project I found a deployment script quietly wiping the directory a web server uses to prove it owns the domain — so the SSL certificate was going to silently fail to renew. We caught it before the certificate lapsed. Nothing was on fire that day. It was just scheduled to be.
If that sounds useful, the fastest start is a free 30-minute call to understand the problem and whether I’m the right fit. No deck, no pitch — and you’re talking to the person who would do the work, since I’m a solo consultant in Phoenix. You can see the rest of what I do at Tech Efficient, or email support@techefficientllc.com. I’ll reply within a day.