télosWing It Research ↗
TELOS EVALUATIONS / UPDATED 28 OCTOBER 2026

Model evaluations

Saved results for five local models across seven suites and 3,147 tasks per model. Scores are checked against individual recorded outcomes.

Saved success rates, not a controlled pass@1 ranking

AFM and Granite include successful selective retries. Models used different token budgets, prompts, and answer extraction. These rates describe the retained reports; they do not establish equal-compute performance or statistical superiority.

ModelSaved successPassed / tasksMarked recoveriesSource
Apple AFM · default, variant unverified62.80%
103 / 164None markedAudited JSON ↓
Ternary Bonsai 4B · 2-bit MLX71.95%
118 / 164None markedAudited JSON ↓
IBM Granite 4.2 3B60.37%
99 / 164None markedAudited JSON ↓
LiquidAI LFM 2.5 8B A1B45.73%
75 / 164None markedAudited JSON ↓
Gemma 4 E4B · 4-bit MLX82.32%
135 / 164None markedAudited JSON ↓

What these suites measure

HumanEval uses recorded Python execution outcomes. Tool-use is Telos’s custom 30-task tool-name and argument-matching suite, not an official BFCL result. Cybersecurity is a 50-task local CWE and remediation-pattern rubric, not a comprehensive security assessment. MMLU and MATH are selections from their parent benchmarks.

Retries and generation budgets

AFM has 28 GPQA and 49 MATH outcomes marked as successful 4,096-token recoveries. Granite has 35 GPQA and 204 MATH recoveries, plus 16 MMLU and 572 ARC repair markers. Markers do not count all attempts; “None marked” does not establish a single attempt. The repair marker’s name mentions 16,000 tokens, but the retained script requests 4,096.

Gemma and Bonsai configurations specify 1,024 tokens for ARC; 2,048 for MMLU, HumanEval and tool-use; 8,192 for GPQA and MATH; and 4,096 for cybersecurity. Adapter behavior also differs. Historical confidence intervals are omitted because selective retries and differing protocols prevent treating them as controlled comparisons.

Model identity

Apple’s run used SystemLanguageModel.default. Its historical “AFM 3 Core Advanced” harness label does not verify the Core/Core Advanced variant. Bonsai’s loaded-model metadata records 2-bit MLX, and Gemma’s records 4-bit. Granite and LFM names are retained run labels; exact checkpoint revisions are not established here.

28 October correction

Bonsai’s master summary disagreed with five detailed reports. Corrected rates are HumanEval 71.95%, GPQA 26.77%, MMLU 62.55%, MATH 56.71%, and ARC 54.61%. Detailed outcomes now govern this page. No new model inference was run.

Source provenance

Audited on 28 October 2026: 35 reports and 15,735 outcomes, with matching task-ID sets within each suite. Downloads contain aggregate results and source hashes; task outcomes exclude prompts and completions.

Apple AFM · default, variant unverified

Audited model summary

ea943dda87db55378e0fadd30207b933ebbe3baa3214b060e09c5f06f2ea6519
Ternary Bonsai 4B · 2-bit MLX

Audited model summary

979b0bf20bbacee77d05eab429811696abae5f476b17fb3c91be53c346fc3167
IBM Granite 4.2 3B

Audited model summary

1cf5efe572d99420ca2990ea29f09fadf3f7aaf3b3b45b5fab8bd164a354cb81
LiquidAI LFM 2.5 8B A1B

Audited model summary

a8b7d8dc682eddd8f17f44f7688faac91a5fcfa09b10f1cb5d2d487b82b12b2d
Gemma 4 E4B · 4-bit MLX

Audited model summary

fda1b38e9657ba294d85d17bc7eab83b28bbb37d5cd54f5bbf6930a8f7633013