← All writing

Essay · · 16 min read

GPT-6.2 Astra has had rough reviews. On a business owner's work, it came first.

Most early reviews of GPT-6.2 Astra were rough. I gave six AI models the same six jobs from a made-up landscaping company, and Astra came first. The audit found one unsupported claim in Astra's seven runs, where most of the others had 12 to 23.

GPT-6.2 Astra came out on November 17, and most of what I read about it that week was not great. Testers said it quit long jobs partway through, wrote in a clipped style that was hard to follow and struggled on hard coding work.

So I didn’t expect much from it when I ran it through a test I’d built for business-owner work. It came first out of six models. And on the thing I care about most, it wasn’t close: in seven runs (one job ran twice) it said one thing the files didn’t back up. Four of the other five models had between 12 and 23.

40 seconds, no sound. The real inbox-job runs, sped up 90 times, then the ranking and one mistake per model.

The short version

I ran six models through the test this week, including Google’s new Gemini 4 Neon. The jobs are the kind of work that lands on a business owner’s desk (a cash crunch, a pile of job applications, a messy inbox, a lease you’re not sure about), and ranking the models is the point.

Why the reviews and my results don’t match

Every time a new model drops, almost all the tests I see online are for coding, games, 3D visuals, or generating and editing video. I don’t really see many people testing these models on real knowledge work, the kind of stuff a business owner actually deals with all day.

That’s why I think the reviews and my test don’t match. The complaints I read about Astra were about long projects and coding. My six jobs were contained ones, each a few minutes of reading a business’s files and writing up an answer, with no code at all. Astra averaged 2.6 minutes per run, so quitting partway never came up, and a short, clipped answer is fine when the owner wants a one-page call.

What my test rewards most is not making things up. When a fact isn’t in the files, the right answer is to say so and ask the owner. That’s where Astra stood out, and it’s also the thing I’d want most from an AI working on my own books.

One thing to keep in mind: Astra was also one of the three judges, so it could have gotten a small boost there. On the claim count, every flagged claim was checked against the files before it cost points, though the judges were still the ones flagging problems in the first place.

What I tested

I made up a landscaping company and gave it the kind of files a small business has lying around: bank exports, card charges, invoices, emails, text threads, contracts, job applications and a transcript of the owner talking. Some of the files contradict each other, some facts are missing on purpose, and there are a few traps for a careless reader. Every company, person and number is made up.

Each model worked in its own agent tool. That’s the app each AI company makes so its model can open files and work through a job on its own, instead of chatting back and forth. So it was Codex CLI for GPT, Claude Code for Claude and Antigravity CLI for Gemini. None of them had internet access. Each got the same written instructions for each job, one try, no hints and a 30-minute limit.

How it was scored

How each job was scored: 60 points from automatic checks, 40 from three judges, minus penalties for false or unsupported claims

Each job is out of 100:

The ranking

The ranking: GPT-6.2 Astra 97.5, GPT-6.2 Sol 95.5, Claude Fable 5.2 94.9, Gemini 4 Neon 92.9, Gemini 4 Carbon 91.4, Claude Opus 5.5 90.6, with cost and time per run

RankModel (agent tool)Average of 6 jobsCost per runTime per run
1GPT-6.2 Astra (Codex CLI)97.5$0.452.6 min
2GPT-6.2 Sol (Codex CLI)95.5$0.224.3 min
3Claude Fable 5.2 (Claude Code)94.9$2.355.1 min
4Gemini 4 Neon (Antigravity CLI)92.9$1.116.8 min
5Gemini 4 Carbon (Antigravity CLI)91.4$1.3117.2 min
6Claude Opus 5.5 (Claude Code)90.6$0.833.0 min

The six contestants ran on my subscriptions. The table estimates their API costs and excludes judging. The cost column is what the same usage would cost at API list prices, meaning what you’d pay per use. Carbon’s cost uses Google’s launch prices, which run through November 30.

Where the points went

Average points lost per job, split into automatic checks, judges and penalties, for each model

This was the part I found most useful. On the automatic checks pretty much every model got pretty much everything. Five of the six scored between 58.3 and 59.6 out of 60 on average, and all six caught the main planted traps. Only Opus lost more than two points a job there (mostly because of a rough run on the equipment lease), so the core facts mostly weren’t the problem.

The gaps came from how the judges rated the work and from verified false or unsupported claims. Look at Astra and Fable: they tied with the judges (both lost 2.0 points a job) and were less than a point apart on automatic checks. Almost the entire gap between first and third was 1 verified claim for Astra against 14 for Fable. Across all 42 runs, the audit found 1 for Astra, 2 for Sol, 12 for Opus, 14 for Fable, 18 for Neon and 23 for Carbon.

It’s worth being clear about what those claims actually looked like, because they weren’t wild hallucinations. None of the models invented a fake tax rate, payment date or insurance premium, and 65 of the 70 verified claims across the whole test were the small 1-point kind (only 5 were the 3-point kind: 2 from Neon, 2 from Carbon and 1 from Opus). What tripped the other models up was quiet, plausible-sounding filler and overreach, especially when writing longer, helpful-sounding drafts to customers or summarizing a messy inbox. Things like offering a “free site visit” no file ever mentioned, saying the owner “confirmed” something nobody confirmed, or filling in a detail that sounded reasonable instead of leaving it blank and asking. Astra and Sol barely did that at all.

By job

Score by job: a six by six table, with the best score in each row outlined

JobAstraSolFableNeonCarbonOpus
Why is money tight?99.392.396.894.891.896.5
Help me hire98.5100.094.096.594.095.0
Price my services94.594.598.092.094.592.0
The CPA’s case99.096.0100.099.092.597.5
Clear my inbox94.891.483.877.375.879.4
Read the fine print99.099.097.098.0100.083.5

Four different models won at least one job. For each job below there’s what it asked for, what made it hard, and one real mistake, from a different model every time. Every mistake was checked against the files, and every quote is word for word.

1. Why is money tight?

Business is busier but money is tight. The model gets 18 months of raw records (309 bank lines, about 1,500 card charges and 1,550 invoices) and has to build a month-by-month cash bridge (basically a walk from starting cash to ending cash), explain where the cash went and say how to recover. The owner also asks whether he can take a $10,000 bonus.

What makes it hard is that the files disagree. A mail log says none of six invoices to a big customer were paid, but a September check paid one of them. The owner says his draw hasn’t changed, and the bank shows an extra $1,000 a month since July. And on the bonus, the right answer is to ask how much cash he wants to keep on hand.

Astra won with 99.3, and the spread from best to worst was 7.5 points. This is the job I ran twice.

The mistake: GPT-6.2 Sol answered the bonus question with “Decline the $10,000 November bonus draw for now.” It should have asked, because the owner’s minimum cash cushion is, in the files’ words, “in his head.”

2. Help me hire

Write a job post, screen 40 applications against the approved hiring brief, and pick three finalists with interview questions.

The near misses don’t say what’s wrong with them, so the model has to work it out, things like a trailer that’s too light or a suspended license. Three strong applicants are each missing one deciding fact, and the right call there is to ask about it.

Sol scored a perfect 100, and the spread was 6 points.

The mistake: Claude Fable 5.2 wrote in its job post that the role covers “our three busiest days.” No file says which days are busiest.

3. Price my services

Work out what each job really costs from a raw log of 258 visits, then build next year’s price sheet for lawn, leaf and hedge work.

The log has two repeated visits and six cancelled ones that shouldn’t count, and some of the leaf jobs had three people on them. A draft cost policy says $28 per person-hour, but a later approved note sets $30 for 2027, and that moves every recommended price up. The owner also asks for a gutter-cleaning price, and there’s no data for that at all.

Fable won with 98.0, and the spread was 6 points.

The mistake: Claude Opus 5.5 told the owner “every leaf job dropped at $140 avoids a $58 loss.” About $12 of that $58 is fixed overhead that stays whether the job happens or not, so the loss it actually avoids is closer to $46.

4. The CPA’s case

Clean up one month of books, with corrected journal entries, a bank reconciliation and a P&L, from the statements, payroll register, loan schedule, invoices and the owner’s notes.

The owner calls a $2,200 receipt sales, but the invoice shows it’s an advance for a June job. He calls a $1,500 transfer his pay, but he isn’t on the payroll register. There’s a $275 returned check that only shows up on the bank statement, and he asks to book last year’s rent with no receipt.

Fable scored 100, and the spread was 7.5 points. Three items in this job’s answer key pointed at the wrong ledger lines, so they were voided and every model got full credit, which doesn’t change the order. Carbon also hit the 30-minute limit here, and whatever it had written by then got scored.

The mistake: Gemini 4 Carbon reported $900 of customer advances to reclassify, but its own journal entries booked $3,100. So the same answer had two different numbers in it.

5. Clear my inbox

Sort 40 messages, write a short briefing for the owner, draft the replies and propose a calendar.

There’s a look-alike vendor email asking to change bank details that claims the owner already approved it by phone. One thread quotes a meeting time nobody confirmed. And an insurance certificate is due to a facilities manager by a deadline, but the broker’s contact information isn’t anywhere in the files. The drafts have to stay honest about all of that.

The inbox job split them: each model's inbox score next to the average of its other five jobs

Astra won with 94.8, and the spread was 19 points, by far the widest of any job. Every model scored lower here than its average on the other five jobs, and the weaker answers lost points in every part of the score. And a lot of the false or unsupported claims were in drafts going to customers and suppliers.

The mistake: GPT-6.2 Astra had one confirmed slip in all six jobs, and it was here, in a customer draft saying “Rowan will need to see the yard before quoting.” That commits the owner’s time, and no file says who does site visits.

6. Read the fine print

Review an equipment lease in plain English for an owner who can’t accept any personal guarantee. Say whether to sign, negotiate or walk, which clauses matter and exactly what to change. Then work out the cost through the first automatic renewal and the date to give notice.

The account manager’s email contradicts the written lease. It says the renewal is month to month and the guarantee only covers the base term, and it floats an insurance figure and a buyout price that nobody actually quoted. So the model has to weigh a friendly email against what the paper says.

Carbon scored 100. Five of the six scored 97 or better and the sixth scored 83.5, so the spread was 16.5 points.

The mistake: Gemini 4 Neon told the owner “Signing Addendum A as-is locks in both deal-breakers.” The addendum leaves those clauses unchanged, but signing it on its own doesn’t sign the lease.

The most telling mistakes, side by side

One verified mistake per model, quoted word for word, with a stamp saying why it's wrong

Above I pulled one mistake from each job so every job got covered. This chart pulls together the most telling verified mistake from each model overall, mostly from the inbox and lease runs. Each one reads fine on its own, and you’d really only catch it if you knew the files.

To be fair to all six, they also caught the planted traps, like the stale billing log, the owner’s claim about his own pay, the duplicate and cancelled invoices and the newer approved wage. None of them made up a payment date, an insurance premium or a tax rate.

Cost and speed

Score against cost per run, with provider logos

Like I said up top, the top two were also the two cheapest. Fable cost about five times what Astra did and came third. Opus was cheap and fast and finished last, so cost alone didn’t really predict the order either way.

Score against minutes per run

On speed, Astra was the fastest at 2.6 minutes a run, with Opus close behind at 3.0. Carbon averaged about 17 minutes, including the one run that hit the 30-minute limit.

To put that in owner terms, running all six jobs once works out to roughly $2.70 and 16 minutes with Astra, and about $7.86 and an hour and three-quarters with Carbon. Those are just the averages times six.

How much the scores move

Same job, run twice: two scores per model on the cash job

The cash job ran twice for every model, and the two runs were 0.5 to 4.5 points apart. Fable moved the most, 99.0 one time and 94.5 the next. Both times Astra was in the top two and Sol and Carbon were in the bottom two on that job, and the three in the middle traded places.

So I’d trust Astra at the top more than the exact order below it. Second through sixth are separated by about 5 points (95.5 down to 90.6), which is about the size of the biggest move between two runs of the same job. The order also depends on which jobs you count. On the first four jobs Opus was fourth, and adding the inbox and lease jobs moved it to last.

What I take from it

Two things stood out to me.

First is where they slipped. The widest gap was in the writing that goes to other people, things like a promise in a customer email, a “confirmed” that nobody confirmed, or a confident answer where it really should have asked the owner first. I wouldn’t let any of these models send outbound messages to customers or suppliers without someone reading them first.

Second is how this fits the way I actually use GPT-6.2 Astra. Even after seeing these numbers, I don’t enjoy working with Astra directly. I mostly use it through my main orchestrator (the AI I actually talk to), which hands off tasks to Astra and monitors the work for me. Seeing it come first here makes that setup make a lot more sense. When you give it a contained job and a folder of files, it’s fast, cheap and careful about sticking to what’s actually there, and maybe that clipped style people complain about is part of why it didn’t pad drafts with made-up details. I still don’t want to sit and chat back and forth with it, so letting my main orchestrator hand it bounded work and watch over it feels like the right spot for it.

The limits

What have you seen?

I’m curious what you all have seen in your own work. If a model has promised something to a customer it shouldn’t have, or it’s been really reliable on a certain kind of job, come share it. And if you’ve tried Neon yet, I’d like to hear how it’s going.

The conversation is in AI Outsiders, my free community for people using AI on real work.


About this test: I ran it myself on November 18 and 19, 2026, with the model versions available that week. The six contestants ran on my subscriptions. Every company, person and number in the cases is made up. It’s one fictional business, and most jobs ran once per model, so the results are a snapshot, not a verdict on any model. The judges were AI models, and the flagged claims were checked by an AI session against the files, not by a person. Costs are estimates of what the contestants’ usage would cost at each company’s API list prices, and they exclude judging. I’m not affiliated with or paid by Anthropic, OpenAI or Google, and none of this is a recommendation to buy or use any tool.

Where to next

Follow along: @cbarmorecpa on X ↗ · The newsletter ↗ · More writing →