Code reviewing lone lisp with Sol and Fable

I've been a solo programmer for over ten years. It is so wonderful to be able to work on my own projects with no pressure of any kind, and just make real the things I envision with my mind. It's always been a recreational, even spiritual, activity for me.

I definitely feel the isolation though. Writing code for no one in particular, just for myself. Never following anyone else's standards. Never knowing what my peers thought. Did I even have peers?

LLMs changed that.

I paid for my Anthropic subscription in early 2026. Chose Anthropic just because I had vaguely heard it was the more ethical of the AI companies. The first thing I did was connect my GitHub account and synchronize all my repositories, then ask Claude Opus 4.6 for a full code review of lone lisp. I was immediately impressed, and code review became a mainstay of my hobby software development ever since. I had a coding buddy now.

The ultimate code review

Anthropic had already released its newest, most powerful model, Claude Mythos Preview, as far back as April this year. Only a few "trusted" corporations and institutions were given access at first, and peasants such as myself needed not apply.

A version of Mythos called Fable was finally released to the public on June 9th, and I began a complete code review of lone lisp on June 10th, the very next day. I wanted to see what the world's smartest AI model had to say about lone lisp.

That was the beginning of the end of my business relationship with Anthropic. Fable launched with incredibly obnoxious safeguards that added enormous friction. It'd often downgrade to Opus while reviewing lone, especially the parts that needed it the most! In spite of these safeguards, the US government imposed export controls on June 12th, causing Anthropic to suspend access. Service was only restored on July 1st.

Anthropic is extremely stingy with Fable. There is a separate meter for it, and on subscriptions it has higher multipliers compared to API prices. One 5h code review window would take out 10% of my weekly quota, and that's the happy path where Fable wouldn't downgrade to Opus.

I nevertheless slogged through this review out of sheer stubbornness. It took over a month to complete this code review! My subscription ended just in time...

Subscribed to OpenAI that same day, and the grass turned out to be much greener here. Sol had just come out, and I had started to wonder if it was really as good as Fable... Let's see!

Fable's invocation numbers are higher due to Anthropic's very low usage limits, as well as cybersecurity blocks and their associated retries.

It costs two thousand two hundred dollars to review lone lisp... For twelve weeks.

Thank god we have subscriptions.

The review skill

I used my own scrutinize parallel code review skill which works by applying "critics" to "subsystems".

The skill currently defines fourteen "critics", which are agent personas that review the code from various perspectives. Twelve were used during this review.

The review coordinator explores the repository and subdivides it into subsystems suitable for review. This limits the scope and keeps context manageable.

12 critics multiplied by 21 subsystems results in a 252 cell code review matrix. Multiplied by the two frontier models, the result is 504 individual review cells. It's not exactly a controlled experiment, but I think it still turned out pretty good.

All reviewers go in blind with no prior context outside of their prompts. No list of known issues is provided.

Agents run in parallel and commit their findings into an orphan ai branch in the git repository. This easily burns even the most generous token allowances!

A filesystem journal style algorithm was used to deal with the interruptions caused by the five hour limits: agents would keep a journal and redo operations that got interrupted and didn't complete. To deal with safety lockouts, the skill also encoded a GNU ddrescue style algorithm: complete all easy cells first, then scrape any blocked cells by retrying with modified prompts until they succeed. This squeezed every last drop of juice out of Fable even in spite of Anthropic's safety classifiers.

These agents produced a lot of output. One finding can contain several problems, and multiple agents can find the same issues independently. When done, the review coordinator goes through the findings and semantically splits and deduplicates them all. Multiple independent findings increase confidence. Agents classify the severity of findings and then results converge into what's essentially a titanic TODO list of issues.

It's these results that I'm going to use as a measure of the performance of Sol and Fable.

The results

The agents produced 538 HIGH/CRITICAL findings which were split into 637 atomic claims and further deduplicated into 232 unique semantic items, one of which was later rejected. I excluded 63 missing test findings, 10 maintainability or duplication recommendations and 5 documentation findings from the tables. This left me with 153 unique defects.

Sol found all four unique defects that Fable classified as CRITICAL. It just happened to classify two of them as HIGH instead. Conversely, Fable also found defects that Sol classified as CRITICAL, but classified one of them as LOW. Their union contains five unique defects, all shared between both code reviews.

Semantic overlap

How many defects were found by both models? How many were unique to each model?

Sol corroborated 109 of Fable's 119 defects, or 91.60%, but missed 10. Fable corroborated 109 of Sol's 143 defects, or 76.22%, but missed 34.

Known issues rediscovered

The project already had an independent list of known issues before either code review. Neither model saw it. After the review was completed, I compared the findings against the 106 pre-existing issues on the list, all of which were still present in the source code.

The observed differences were not statistically significant. Paired McNemar test p = 0.503.

The grass wasn't always greener

Anthropic became famous for its incredibly sensitive Fable safety classifier, but OpenAI turned out to not be much better. Security reviews mostly worked, but robustness reviews hit walls immediately.

For Fable, a cybersecurity failure is inferred from downgrades to Opus. The skill directs the model to detect this and abort rather than continue reviewing with a weaker model. For Sol, it is a cyber_policy termination. General unavailability comprises usage exhaustion, timeouts and overloaded compute capacity.

The security/robustness prompts already differed initially, and diverged further during the code review due to retries. Both models eventually produced finished review output for all 42 defensive code review cells after enough retries. Exploit vocabulary had to be beaten out of the models as though the words were unforgivable wrongthink. This means the reports were less directly comparable, but the first attempt table is still a good measure of how much friction these safeguards add.

These 42 defensive assignments were only one sixth of the review matrix, but produced 24 of OpenAI's 36 cyber_policy blocks: exactly two thirds.

They weren't the only ones though. There were, in total, 50 downgrade events across Fable's 702 attempts and 36 cybersecurity blocks across Sol's 257 attempts, including their retries.

The high number of events in correctness and rigor is highly disappointing, since these are the most valuable code reviewers.

There were far too many events in important subsystems that include the memory allocator, garbage collector, I/O and module system. Fable especially did not like the ELF patcher.

Discussion

Fable meticulously catalogues stuff. Sol synthesizes and triages instead. Fable output nearly three times more tokens than Sol, 1,763 versus 1,091 entries, as well as 373 versus 78 INFO/NOTE observations.

Both found the same critical issues in lone lisp! This is an incredible result, since it suggests OpenAI has extremely competitive performance where it counts at around 30% of the API dollar cost!

Arguably, OpenAI turned out to be just as locked down as Anthropic if not more when it comes to cybersecurity code reviews. Robustness cells were blocked straight up in almost all cases!

Anthropic imposed the larger capacity and orchestration burden overall. OpenAI imposed more blocks on defensive prompts, especially in robustness critics.

Using Fable for defensive cybersecurity was miserable. Anthropic accepted me into their cybersecurity program, but it didn't help at all with my code review work.

Sol is just as difficult to work with. OpenAI tells users to verify their identity and apply to their TAC program. I tried it, but was apparently rejected before any verification. cyber_verification_precheck_failed, their servers say. My account is not eligible. No explanations, no appeals, nothing. There were no retries at first, but after a while it started allowing me to verify. I did so, but still failed to get into the program.

Even with all these cybersecurity setbacks, OpenAI is still providing a staggering amount of value with their subscriptions. Their plans are actually honest, and the sheer number of usage resets they're giving out is just insane. My research into other services suggests a Kimi K3 subscription is the best alternative to frontier labs right now: the terms of service are the most reasonable and most friendly to authorized defensive cybersecurity work. Sadly, new subscriptions are paused at the moment. Looks like OpenAI's my home for the foreseeable future.

Fabula est acta.