Claude Computer Use: What It Reaches When Your App Has No API
- Branden Bell
- 2 days ago
- 5 min read
Somebody asked me last week whether Claude could operate the quoting tool their office has run since 2014. No API, no export, a Windows program that opens on one machine. The answer is yes, and the part worth knowing is that Anthropic would rather it didn't.
Computer use is where Claude takes screenshots of your desktop and clicks and types the way you would. It's the one that demos well. It's also the one Anthropic ranks last in its own documentation, and the reason it ranks last tells you more about when to reach for it than any demo does. Verified against Anthropic's own pages, August 2026.
The order Claude actually tries
Claude uses the most precise tool it has and falls back to your screen only when nothing else reaches. Anthropic publishes that ladder on both surfaces.
The Cowork help article lists three rungs: a connector first, then the Chrome browser, then screen interaction. The reasoning isn't philosophical.
Claude prioritizes the fastest method. For example, pulling messages through your Slack connection takes seconds, but navigating Slack through your screen takes much longer and is more error-prone.
The Claude Code CLI docs cover a different surface and say it harder, with a rung added for shell commands:
Computer use is the broadest and slowest, so Claude tries the most precise tool first.
And on what's left at the bottom of that ladder:
Screen control is reserved for things nothing else can reach: native apps, simulators such as the iOS Simulator, and tools without an API.
That's the honest scope, and it's the one the vendor sells. If your tool has an API or a connector, you don't want screen control, you want the connector.
Every other app gets the top tier
Claude's level of control isn't uniform, and the tiers are set by app category rather than by what you asked for. The CLI docs spell it out: browsers and trading platforms are view-only, terminals and IDEs are click-only, and everything else gets full control.
Read that last clause slowly. The category everybody means when they say "a desktop app with no API" isn't a special case with extra handling. It sits in "everything else." Your accounting software, your CRM's desktop client, your practice management tool, the file manager: full control, once approved.
Approval itself is narrower than people assume, and that part is good news. Claude asks before touching each application, and on the CLI those approvals last for the current session only. I covered how that fits the wider Cowork boundary in what Cowork can reach when you connect a folder.
Where the per-app approval stops reaching
Here's the sentence I'd want a client to read before turning any of this on. It's Anthropic's, in the safety section of its own Cowork article:
Note that actions taken in one app can impact other apps. For example, clicking a link in your email app might open it in Chrome, even if you haven't explicitly granted Claude permission to use Chrome (we can prevent Claude from seeing the Chrome window but can't stop the link from opening).
Per-app approval is a permission model, not a containment model. It governs what Claude can see and drive. It doesn't govern what the app you approved can do to the rest of your machine when Claude clicks something inside it. Those are two different problems, and the approval prompt only covers the first one.
Credit where it's earned, because the engineering aimed at the approved app is real. The terminal is excluded from screenshots so Claude never reads its own output, and the Esc key aborts from anywhere with the keypress consumed, so a prompt injection can't use it to dismiss a dialog. Three apps carry an extra warning before you approve them: terminals and IDEs as equivalent to shell access, Finder as able to read or write any file, System Settings as able to change system settings. The docs don't dress up what that warning does.
These apps aren't blocked. The warning lets you decide whether the task warrants that level of access.
So it's your call, in a prompt, mid-task, while you're trying to get something done. Anthropic says the same about the rest of it, and I haven't seen another vendor put it this plainly:
These guardrails are part of how Claude is trained and instructed, but they aren't absolute. Don't rely on them as a substitute for blocking access to sensitive apps.
What went wrong inside granted access
I've watched the expensive version of this. Rebuilding my own site with Claude, the editor's focus quietly drifted to a different page in the middle of a batch, three removal calls landed on the wrong container, and a finished About page was wiped. Every edit verified as landed. The check that ran right after came back clean, because an empty page has no defects left to find.
Nothing was breached. Every action happened inside access I'd granted, on an app I'd approved, doing the category of thing I'd asked for. The permission model worked exactly as designed and I still lost the page. That build and the focus guard that came out of it are written up on the Wix page builder build page, and the skill is a free download.
The rules I wrote down
What I do on my own machine, and what I set up with clients:
Close it before you start it: Claude screenshots whatever is visible, so anything sensitive on screen goes into the transcript.
Grant the app, then ask what that app can open: email clients, anything with a link, and anything with a share menu reach the furthest past their own window.
Keep it away from financial, medical and legal work: that's Anthropic's own recommendation, and there's no upside in testing it.
Make it report what it did, not just that it finished: a run that says "done" doesn't tell you what it touched.
One more thing to plan around: computer use is a research preview on Pro and Max plans only, with no Team or Enterprise access, and your computer has to be awake with the desktop app open. If you want the browser instead of the desktop, the scoping rules are different and I wrote those up in how Claude in Chrome permissions work. For where this sits in the wider picture, start with how to use Claude in 2026.
Sources
Working out which rung of that ladder your work belongs on is most of the job, and it's cheaper to get right before anything gets built. If you want a second opinion, the intro call is fifteen to twenty minutes and costs nothing. After that it's $250 for the first hour and $75 an hour, with the hours estimated up front. Whatever gets built runs in your Claude account and you own it.
Comments