Smart contracts are permanent once deployed, which means code bugs can lead straight to major financial loss. Many crypto projects now try to use AI models to catch these vulnerabilities before going live. I came across a new study that tested whether bug hunting skills are a general trait of AI or just a quirk of specific model designs. It turns out that assuming every AI model behaves the same way is a bit of a trap.
The researchers set out to learn if reported AI vulnerability detection patterns transfer across different model architectures. They first replicated an earlier study by Chen et al. on GPT-4 and confirmed its high-recall, low-precision behavior. Then they tested other designs, including Mixture-of-Experts and RNN-like models, alongside a preliminary look at State Space Models. They found that this detection pattern does not transfer uniformly to all models. Interestingly, GPT-4 and Gemma-3 shared the same detection profile despite a gap of roughly two orders of magnitude in parameter counts. Meanwhile, two Mixture-of-Experts models from different lineages showed divergent behaviors. This proves that architectural family alone is insufficient to predict how an AI model will perform at catching smart contract bugs.
Curious about the details? Look up When LLM Architectures Meet Smart Contract Vulnerability Detection: a Replication Study at arxiv.org/abs/2610.38621. The authors are Gerardo Iuliano, Gabriele De Vito and Dario Di Nucci.
This is not financial advice. The information provided is for educational and informational purposes only.
The cover image was generated by AI.