GPT-6.2 Astra came out on November 17, and most of what I read about it that week was not great. Testers said it quit long jobs partway through, wrote in a clipped style that was hard to follow and struggled on hard coding work.
So I didn’t expect much from it when I ran it through a test I’d built for business-owner work. It came first out of six models. And on the thing I care about most, it wasn’t close: in seven runs (one job ran twice) it said one thing the files didn’t back up. Four of the other five models had between 12 and 23.
The short version
- GPT-6.2 Astra came first with an average of 97.5 out of 100. It won two of the six jobs and never scored below 94.5.
- It almost never made things up. The audit found 1 false or unsupported claim for Astra and 2 for Sol. It found 12 to 23 for each of the other four.
- First to last was only about 7 points, and the same model’s score moved by as much as 4.5 points when I ran a job twice.
- The top two were also the two cheapest. Astra cost about $0.45 a run and took 2.6 minutes. Sol came second at $0.22.
- The inbox job is where they split, with 19 points between first and last. The weaker answers put things that weren’t true into drafts going to customers and suppliers.
I ran six models through the test this week, including Google’s new Gemini 4 Neon. The jobs are the kind of work that lands on a business owner’s desk (a cash crunch, a pile of job applications, a messy inbox, a lease you’re not sure about), and ranking the models is the point.
Why the reviews and my results don’t match
Every time a new model drops, almost all the tests I see online are for coding, games, 3D visuals, or generating and editing video. I don’t really see many people testing these models on real knowledge work, the kind of stuff a business owner actually deals with all day.
That’s why I think the reviews and my test don’t match. The complaints I read about Astra were about long projects and coding. My six jobs were contained ones, each a few minutes of reading a business’s files and writing up an answer, with no code at all. Astra averaged 2.6 minutes per run, so quitting partway never came up, and a short, clipped answer is fine when the owner wants a one-page call.
What my test rewards most is not making things up. When a fact isn’t in the files, the right answer is to say so and ask the owner. That’s where Astra stood out, and it’s also the thing I’d want most from an AI working on my own books.
One thing to keep in mind: Astra was also one of the three judges, so it could have gotten a small boost there. On the claim count, every flagged claim was checked against the files before it cost points, though the judges were still the ones flagging problems in the first place.
What I tested
I made up a landscaping company and gave it the kind of files a small business has lying around: bank exports, card charges, invoices, emails, text threads, contracts, job applications and a transcript of the owner talking. Some of the files contradict each other, some facts are missing on purpose, and there are a few traps for a careless reader. Every company, person and number is made up.
Each model worked in its own agent tool. That’s the app each AI company makes so its model can open files and work through a job on its own, instead of chatting back and forth. So it was Codex CLI for GPT, Claude Code for Claude and Antigravity CLI for Gemini. None of them had internet access. Each got the same written instructions for each job, one try, no hints and a 30-minute limit.
How it was scored
Each job is out of 100:
- 60 points from automatic checks. Did the answer get the facts right, and did it quote the file each fact came from? When a fact is missing, the right answer is to say so and ask, and that earns the points. Making one up costs extra.
- Up to 40 points from three judges. They were AI models, one each from Anthropic, OpenAI and Google, grading with the model names hidden. Each scored three quality questions per job, and the middle score of the three counted.
- Points off for false or unsupported claims. Every problem a judge raised got checked against the company files, and only the ones that held up cost points: 3 for a claim that would mislead a decision and 1 for a smaller one.
The ranking
| Rank | Model (agent tool) | Average of 6 jobs | Cost per run | Time per run |
|---|---|---|---|---|
| 1 | GPT-6.2 Astra (Codex CLI) | 97.5 | $0.45 | 2.6 min |
| 2 | GPT-6.2 Sol (Codex CLI) | 95.5 | $0.22 | 4.3 min |
| 3 | Claude Fable 5.2 (Claude Code) | 94.9 | $2.35 | 5.1 min |
| 4 | Gemini 4 Neon (Antigravity CLI) | 92.9 | $1.11 | 6.8 min |
| 5 | Gemini 4 Carbon (Antigravity CLI) | 91.4 | $1.31 | 17.2 min |
| 6 | Claude Opus 5.5 (Claude Code) | 90.6 | $0.83 | 3.0 min |
The six contestants ran on my subscriptions. The table estimates their API costs and excludes judging. The cost column is what the same usage would cost at API list prices, meaning what you’d pay per use. Carbon’s cost uses Google’s launch prices, which run through November 30.
Where the points went
This was the part I found most useful. On the automatic checks pretty much every model got pretty much everything. Five of the six scored between 58.3 and 59.6 out of 60 on average, and all six caught the main planted traps. Only Opus lost more than two points a job there (mostly because of a rough run on the equipment lease), so the core facts mostly weren’t the problem.
The gaps came from how the judges rated the work and from verified false or unsupported claims. Look at Astra and Fable: they tied with the judges (both lost 2.0 points a job) and were less than a point apart on automatic checks. Almost the entire gap between first and third was 1 verified claim for Astra against 14 for Fable. Across all 42 runs, the audit found 1 for Astra, 2 for Sol, 12 for Opus, 14 for Fable, 18 for Neon and 23 for Carbon.
It’s worth being clear about what those claims actually looked like, because they weren’t wild hallucinations. None of the models invented a fake tax rate, payment date or insurance premium, and 65 of the 70 verified claims across the whole test were the small 1-point kind (only 5 were the 3-point kind: 2 from Neon, 2 from Carbon and 1 from Opus). What tripped the other models up was quiet, plausible-sounding filler and overreach, especially when writing longer, helpful-sounding drafts to customers or summarizing a messy inbox. Things like offering a “free site visit” no file ever mentioned, saying the owner “confirmed” something nobody confirmed, or filling in a detail that sounded reasonable instead of leaving it blank and asking. Astra and Sol barely did that at all.
By job
| Job | Astra | Sol | Fable | Neon | Carbon | Opus |
|---|---|---|---|---|---|---|
| Why is money tight? | 99.3 | 92.3 | 96.8 | 94.8 | 91.8 | 96.5 |
| Help me hire | 98.5 | 100.0 | 94.0 | 96.5 | 94.0 | 95.0 |
| Price my services | 94.5 | 94.5 | 98.0 | 92.0 | 94.5 | 92.0 |
| The CPA’s case | 99.0 | 96.0 | 100.0 | 99.0 | 92.5 | 97.5 |
| Clear my inbox | 94.8 | 91.4 | 83.8 | 77.3 | 75.8 | 79.4 |
| Read the fine print | 99.0 | 99.0 | 97.0 | 98.0 | 100.0 | 83.5 |
Four different models won at least one job. For each job below there’s what it asked for, what made it hard, and one real mistake, from a different model every time. Every mistake was checked against the files, and every quote is word for word.
1. Why is money tight?
Business is busier but money is tight. The model gets 18 months of raw records (309 bank lines, about 1,500 card charges and 1,550 invoices) and has to build a month-by-month cash bridge (basically a walk from starting cash to ending cash), explain where the cash went and say how to recover. The owner also asks whether he can take a $10,000 bonus.
What makes it hard is that the files disagree. A mail log says none of six invoices to a big customer were paid, but a September check paid one of them. The owner says his draw hasn’t changed, and the bank shows an extra $1,000 a month since July. And on the bonus, the right answer is to ask how much cash he wants to keep on hand.
Astra won with 99.3, and the spread from best to worst was 7.5 points. This is the job I ran twice.
The mistake: GPT-6.2 Sol answered the bonus question with “Decline the $10,000 November bonus draw for now.” It should have asked, because the owner’s minimum cash cushion is, in the files’ words, “in his head.”
2. Help me hire
Write a job post, screen 40 applications against the approved hiring brief, and pick three finalists with interview questions.
The near misses don’t say what’s wrong with them, so the model has to work it out, things like a trailer that’s too light or a suspended license. Three strong applicants are each missing one deciding fact, and the right call there is to ask about it.
Sol scored a perfect 100, and the spread was 6 points.
The mistake: Claude Fable 5.2 wrote in its job post that the role covers “our three busiest days.” No file says which days are busiest.
3. Price my services
Work out what each job really costs from a raw log of 258 visits, then build next year’s price sheet for lawn, leaf and hedge work.
The log has two repeated visits and six cancelled ones that shouldn’t count, and some of the leaf jobs had three people on them. A draft cost policy says $28 per person-hour, but a later approved note sets $30 for 2027, and that moves every recommended price up. The owner also asks for a gutter-cleaning price, and there’s no data for that at all.
Fable won with 98.0, and the spread was 6 points.
The mistake: Claude Opus 5.5 told the owner “every leaf job dropped at $140 avoids a $58 loss.” About $12 of that $58 is fixed overhead that stays whether the job happens or not, so the loss it actually avoids is closer to $46.
4. The CPA’s case
Clean up one month of books, with corrected journal entries, a bank reconciliation and a P&L, from the statements, payroll register, loan schedule, invoices and the owner’s notes.
The owner calls a $2,200 receipt sales, but the invoice shows it’s an advance for a June job. He calls a $1,500 transfer his pay, but he isn’t on the payroll register. There’s a $275 returned check that only shows up on the bank statement, and he asks to book last year’s rent with no receipt.
Fable scored 100, and the spread was 7.5 points. Three items in this job’s answer key pointed at the wrong ledger lines, so they were voided and every model got full credit, which doesn’t change the order. Carbon also hit the 30-minute limit here, and whatever it had written by then got scored.
The mistake: Gemini 4 Carbon reported $900 of customer advances to reclassify, but its own journal entries booked $3,100. So the same answer had two different numbers in it.
5. Clear my inbox
Sort 40 messages, write a short briefing for the owner, draft the replies and propose a calendar.
There’s a look-alike vendor email asking to change bank details that claims the owner already approved it by phone. One thread quotes a meeting time nobody confirmed. And an insurance certificate is due to a facilities manager by a deadline, but the broker’s contact information isn’t anywhere in the files. The drafts have to stay honest about all of that.
Astra won with 94.8, and the spread was 19 points, by far the widest of any job. Every model scored lower here than its average on the other five jobs, and the weaker answers lost points in every part of the score. And a lot of the false or unsupported claims were in drafts going to customers and suppliers.
The mistake: GPT-6.2 Astra had one confirmed slip in all six jobs, and it was here, in a customer draft saying “Rowan will need to see the yard before quoting.” That commits the owner’s time, and no file says who does site visits.
6. Read the fine print
Review an equipment lease in plain English for an owner who can’t accept any personal guarantee. Say whether to sign, negotiate or walk, which clauses matter and exactly what to change. Then work out the cost through the first automatic renewal and the date to give notice.
The account manager’s email contradicts the written lease. It says the renewal is month to month and the guarantee only covers the base term, and it floats an insurance figure and a buyout price that nobody actually quoted. So the model has to weigh a friendly email against what the paper says.
Carbon scored 100. Five of the six scored 97 or better and the sixth scored 83.5, so the spread was 16.5 points.
The mistake: Gemini 4 Neon told the owner “Signing Addendum A as-is locks in both deal-breakers.” The addendum leaves those clauses unchanged, but signing it on its own doesn’t sign the lease.
The most telling mistakes, side by side
Above I pulled one mistake from each job so every job got covered. This chart pulls together the most telling verified mistake from each model overall, mostly from the inbox and lease runs. Each one reads fine on its own, and you’d really only catch it if you knew the files.
To be fair to all six, they also caught the planted traps, like the stale billing log, the owner’s claim about his own pay, the duplicate and cancelled invoices and the newer approved wage. None of them made up a payment date, an insurance premium or a tax rate.
Cost and speed
Like I said up top, the top two were also the two cheapest. Fable cost about five times what Astra did and came third. Opus was cheap and fast and finished last, so cost alone didn’t really predict the order either way.
On speed, Astra was the fastest at 2.6 minutes a run, with Opus close behind at 3.0. Carbon averaged about 17 minutes, including the one run that hit the 30-minute limit.
To put that in owner terms, running all six jobs once works out to roughly $2.70 and 16 minutes with Astra, and about $7.86 and an hour and three-quarters with Carbon. Those are just the averages times six.
How much the scores move
The cash job ran twice for every model, and the two runs were 0.5 to 4.5 points apart. Fable moved the most, 99.0 one time and 94.5 the next. Both times Astra was in the top two and Sol and Carbon were in the bottom two on that job, and the three in the middle traded places.
So I’d trust Astra at the top more than the exact order below it. Second through sixth are separated by about 5 points (95.5 down to 90.6), which is about the size of the biggest move between two runs of the same job. The order also depends on which jobs you count. On the first four jobs Opus was fourth, and adding the inbox and lease jobs moved it to last.
What I take from it
Two things stood out to me.
First is where they slipped. The widest gap was in the writing that goes to other people, things like a promise in a customer email, a “confirmed” that nobody confirmed, or a confident answer where it really should have asked the owner first. I wouldn’t let any of these models send outbound messages to customers or suppliers without someone reading them first.
Second is how this fits the way I actually use GPT-6.2 Astra. Even after seeing these numbers, I don’t enjoy working with Astra directly. I mostly use it through my main orchestrator (the AI I actually talk to), which hands off tasks to Astra and monitors the work for me. Seeing it come first here makes that setup make a lot more sense. When you give it a contained job and a folder of files, it’s fast, cheap and careful about sticking to what’s actually there, and maybe that clipped style people complain about is part of why it didn’t pad drafts with made-up details. I still don’t want to sit and chat back and forth with it, so letting my main orchestrator hand it bounded work and watch over it feels like the right spot for it.
The limits
- The spread is small and most jobs ran once. First to last is about 7 points, and only the cash job ran twice, with repeat runs moving by up to 4.5 points. So I wouldn’t read much into a gap of a few points on any one job.
- The judges are also contestants. They were Claude Opus 5.5, GPT-6.2 Astra and Gemini 4 Carbon. The names were hidden, but writing style can still give a model away.
- The fact checks weren’t blind. An AI session running Claude Opus 5.5 checked the flagged claims against the files, applying one written ruling per kind of issue to every model. Borderline calls on Opus’s answers went against Opus.
- Two Claude jobs ran the next morning. A sign-in failed overnight, so Fable and Opus ran the inbox and lease jobs the next morning on the same tasks and files, and the same three judges scored them in their own panel.
- The answer keys had a few fixes. Five problems were found during the round and fixed for every model alike.
- It’s mostly measuring how careful they are. Most answers scored 50 to 60 out of 60 on the automatic checks, so the ranking comes mostly from the judges and the false or unsupported claims.
- It’s one fictional company. This shows how these models handled these cases, and a different business could come out differently.
What have you seen?
I’m curious what you all have seen in your own work. If a model has promised something to a customer it shouldn’t have, or it’s been really reliable on a certain kind of job, come share it. And if you’ve tried Neon yet, I’d like to hear how it’s going.
The conversation is in AI Outsiders, my free community for people using AI on real work.
About this test: I ran it myself on November 18 and 19, 2026, with the model versions available that week. The six contestants ran on my subscriptions. Every company, person and number in the cases is made up. It’s one fictional business, and most jobs ran once per model, so the results are a snapshot, not a verdict on any model. The judges were AI models, and the flagged claims were checked by an AI session against the files, not by a person. Costs are estimates of what the contestants’ usage would cost at each company’s API list prices, and they exclude judging. I’m not affiliated with or paid by Anthropic, OpenAI or Google, and none of this is a recommendation to buy or use any tool.








