Comparison of Models Used for Aviation Briefings

I have a collection of prompts and tools that I give to LLMs to generate daily aviation briefings. The briefings are supposed to be a summary of everything interesting that’s happened in the air in the past several hours, either globally or in a region of interest. The model should contextualize aircraft activity in terms of behavioral analysis, historical patterns, geography, and current events:

The fire west of Piñon Hills stood at roughly 2,700 acres and 0% containment in the latest reports, with cooler overnight conditions helping crews strengthen lines. This morning’s air picture is recon and rotary rather than Saturday’s tanker parade: N8PQ, the Aero Commander 690A, is back overhead at 8,000 ft, K-MAX N107MW is working the fire at 4,200 ft, Bell 205 N33HX is staging at Fox Field, and Tanker 104, an Erickson Aero Tanker MD-87, sits on the ramp at San Bernardino International.

Writing a worthwhile briefing is a challenging task. It’s agentic, with models using 15-20 tools including querying databases, searching the web, and writing and executing scripts to process data. It also requires judgment from a model on what to include in the briefing and how to put together separate pieces of data to tell a story without overstating a conclusion. The agents filter through a large amount of data looking for interesting items, find corroborating sources, take screenshots, and edit it all into a short (less than 6000 characters) report.

I’ve tried Claude Opus 4.8, Fable 5, OpenAI GPT-5.6-Sol, and Kimi K3 on this task. The tl;dr if you’re paying retail API token prices:

• Opus 4.8 is cheap, fast, and its output is good.
• Kimi K3 costs about the same as Opus 4.8 and its output is of similar quality (it’s saved by the fact-checker) but it takes 5-6x as long as Opus 4.8.
• Fable 5 costs 2-5x as much as Opus 4.8 or Kimi K3. I like the feel of its output best, but it needs to be reined in by the fact-checker almost as much as Kimi K3.
• GPT-5.6-Sol is cheap and fast and makes few mistakes, but its briefings are too short and boring.

I think GPT-5.6-Sol is the smartest model available at the moment, but for this task Fable 5 does better, and Opus 4.8 does almost as well for far cheaper.

Some numbers

The initial prompt for the task is about 25 K - 30 K tokens, depending on whether the briefing is for the world or just Southern California. (All token calculations are done with the OpenAI tokenizer; others may be different.)

The custom tools to give the models access to aircraft and geospatial data, and the guides on how to best use them consume another 50 K tokens.

I’ve done six briefings with Fable 5 and two briefings with each of the other models.

The following table lists time and cost for a single briefing, for each model. Time includes the total time for research, review, corrections and posting. Costs are API-equivalent estimates based on current pricing with token caching included.

Model	# Briefings	Median time (minutes)	Median cost
Opus 4.8	2	24.8	$10.06
Fable 5	6	31.5	$30.33
Kimi K3	2	132.4	$10.43
GPT-5.6-Sol	2	40.0	$15.63

After a model writes the first draft of a briefing it sends it and the supporting material to an adversarial fact-checker powered by GPT-5.6-Sol. The model then rewrites the report based on the fact-checker’s findings.

For each briefing I classified the fact-checker’s findings into “major”, more substantive corrections, and “minor”, softer issues. The table below shows the median number of writer tool calls and the mean number of corrections per report for each model.

Model	# Tool calls	Major corrections	Minor corrections
Opus 4.8	66	0.0	4.0
Fable 5	93.5	2.5	7.0
Kimi K3	125	2.5	7.5
GPT-5.6-Sol	116	1.0	2.0

The GPT-5.6-Sol reports used separate GPT-5.6-Sol reviewer sessions, following the same adversarial process as the other models. The review step added a median $1.95 worth of GPT-5.6-Sol tokens to the cost of a report, and review costs are already included in the costs listed above.

Report quality

These are my feelings after reading a few reports.

Opus 4.8

Opus 4.8 did a good job of creating rich reports with strong themes, and it showed good editorial judgment by removing weak leads not supported by further research.

As an example, it might lead with a developing emergency:

N969WR … was cruising at FL450 across the Oklahoma/Texas panhandle when it squawked 7700 at ~2204Z and began a continuous descent, averaging around 2,300 ft/min.

It combined that lead with a large firefighting mobilisation, an EA-37B, an E-6B, allied tanker movements and forward-looking airspace notices. This was varied and interesting without feeling like a list of unrelated aircraft.

The fact-checker didn’t refute any of Opus 4.8’s conclusions, and didn’t find any unsupported claims. Corrections were mostly wording changes to avoid implying unwarranted precision.

Fable 5

Fable 5 produced the most detailed and ambitious reports. It was strongest when it could connect several observations into a larger story.

For example, the 15 July world report described a shared GPS spoofing pattern near Smolensk:

Three airliners (Turkish, Belavia, Air Serbia) each plotted flying an impossible 1.2-nm, 55-kt circle around the same fixed point southwest of Smolensk while MLAT put the real aircraft 400 km away.

The 14 July SoCal report turned an anonymous military track into a strong lead:

A silent visitor working W-291 — ae685e, a US military hex with no callsign, came down from Oregon overnight at FL270 and has flown a broad 21,000-ft circuit off the San Diego/northern Baja coast for hours.

Fable 5 also followed stories across several days. It tracked tanker relays, airlift movements, range activity and unusual aircraft between reports.

Fable 5’s reports were detailed, visual, and interesting. They also needed significant corrections, an average of 2.5 substantive and 7 minor problems per report.

Its main weakness was overprecision. The fact-checker regularly corrected:

• exact start and end times
• track boundaries
• aircraft counts
• claims about every member of a group
• statements that joined events across an anchor time
• mission interpretations that went beyond the observed movement

An example of a significant error (that GPT-5.6-Sol let pass but that I found on review) was its report on a Russian Be-200 amphibious aircraft. Its published heading said:

A Russian EMERCOM Be-200ChS scooping the Donbas.

The report described the aircraft as working “a 300-km chain of fire sites”, but when I saw the map that seemed implausible. I asked Fable 5 about it, and eventually it posted an update:

The evidence now says ‘repeated brief landings at unmapped sites, purpose unknown’.

I still don’t know what that aircraft was doing, but it’s an example of Fable 5 going significantly off course.

Kimi K3

Kimi K3 is really slow, and the fact-checker had to correct it more than any other model, but the quality of its (corrected) output once it was finally done was excellent–as good as Opus 4.8, possibly even Fable 5.

KC-135 declares an emergency over the Irish Sea, home safe at Mildenhall.

Fox Field staged and scattered — seven fire aircraft on the ramp on the fire squawk all morning; the 737 Fireliner and RJ-85 launched ~10am and ferried north without a drop run.

There’s just no reason to use Kimi K3 since it costs as much as Opus 4.8, is about as good, but much slower.

GPT-5.6-Sol

GPT-5.6-Sol shows so much restraint that its briefings are boring: dry, no flavor, and not particularly interesting.

LATTE18 / N650RX, a trustee-registered Global 6500, flew roughly 4 broad clockwise circuits at FL360 from 01:28 to 02:14Z, then departed southwest. The direct ADS-B track measured about 38 by 16 nautical miles. Public aviation reporting associates this tail with SNC’s Army ATHENA-S fleet; SNC confirms its 2 ATHENA aircraft support US Army airborne ISR, but SNC does not name the tails. The same offshore block appears repeatedly in the recent archive.

I generally like GPT-5.6-Sol but its briefings were just too dull.