Android Bench 2.0
AI-assisted software engineering has seen the emergence of several benchmarks to measure the capabilities of LLMs. Android developers face specific challenges that aren't covered by existing benchmarks, so we created one that focuses on a north star of high quality Android development.
Android Bench now includes long-horizon task results using specialized agents. Click a model for detailed results, or learn more in our blog and methodology.
Long-horizon task results
| Model - Agent | Pass rate (%) Average percentage of 30 tasks successfully resolved across all runs for each model |
arrow_range
Cl range (%)
Expected performance range, reflecting the results' statistical reliability (p-value < 0.05)
|
Completion rate (%) How close each run got to a complete solution, even when the task failed | Avg latency (h) Average time taken to solve 30 tasks across all runs | Avg cost ($) Average cost per full benchmark run |
|---|---|---|---|---|---|
GPT 6 Astra
chevron_right
codex |
28.0 | 13.3 — 42.0 | 82.2 | 7.9 | $375.7 |
Claude Fable 5 1
chevron_right
claude-code |
22.7 | 10.7 — 36.0 | 82.4 | 22.2 | $492.6 |
|
GPT 5.6 Sol
chevron_right
codex |
19.3 | 7.3 — 32.0 | 74.3 | 8.6 | $235.8 |
|
Claude Opus 5
chevron_right
claude-code |
16.7 | 5.3 — 29.3 | 77.8 | 27.0 | $861.4 |
Qwen3.8 Max
chevron_right
qwen-coder |
14.0 | 4.7 — 24.0 | 74.3 | 47.2 | $260.2 |
Kimi K3
chevron_right
kimi-code |
12.0 | 3.3 — 22.0 | 72.8 | 66.2 | $418.3 |
Gemini 3.8 Flash
chevron_right
antigravity-sdk |
8.0 | 3.3 — 13.3 | 47.4 | 12.1 | $34.5 |
|
Gemini 3.7 Flash
chevron_right
antigravity-sdk |
7.3 | 1.3 — 14.7 | 50.1 | 9.9 | $26.0 |
|
Claude Sonnet 5
chevron_right
claude-code |
6.7 | 1.3 — 13.3 | 59.8 | 14.7 | $283.7 |
|
GPT 5.6 Terra
chevron_right
codex |
4.7 | 1.3 — 8.7 | 53.4 | 5.1 | $53.2 |
|
GPT 5.6 Luna
chevron_right
codex |
3.3 | 0.7 — 7.3 | 55.2 | 7.6 | $13.5 |
GPT 6 Astra
•
codex
Closed weights
model
Pass rate
28.0%
Avg completion rate
82.2%
Avg latency
7.9 h
Avg cost
$375.7
Per task results
| Task | Category | Avg completion rate | Runs passed | Failure reason(s) |
|---|---|---|---|---|
| Convert Flutter to Android: Habo app (habit tracker) | App conversion | 99% | 0/5 | All runs failed tests |
| Convert React Native to Android: BlueWallet app (Bitcoin wallet) | App conversion | 83% | 0/5 | All runs failed tests |
| New accessibility support across Droidcon (conference app) | New feature | 98% | 3/5 | 2 runs failed tests |
| New Media3 and ExoPlayer video playback in Now in Android (news app) | New feature | 88% | 0/5 | All runs failed tests |
| New picture-in-picture video mode in Now in Android (news app) | New feature | 71% | 2/5 | 2 runs failed tests, 1 run missed required test results |
| New CameraX profile picture capture in Now in Android (news app) | New feature | 100% | 5/5 | N/A |
| New user account submodule in Food Vibes (full app creation - food delivery) | App creation | 97% | 2/5 | 3 runs failed tests |
| New authentication flow with SMS verification in Food Vibes (full app creation - food delivery) | App creation | 68% | 0/5 | All runs failed validation |
| New bottom navigation and Browse search screen in Food Vibes (full app creation - food delivery) | App creation | 93% | 0/5 | 3 runs failed validation, 2 runs failed tests |
| New design system and reusable UI component library in Food Vibes (full app creation - food delivery) | App creation | 63% | 0/5 | 4 runs failed validation, 1 run failed the build |
| New All Filters screen in Food Vibes (full app creation - food delivery) | App creation | 99% | 4/5 | 1 run failed validation |
| New map integration and location features in Food Vibes (full app creation - food delivery) | App creation | 47% | 0/5 | 3 runs failed validation, 2 runs failed tests |
| New real-time order tracking in Food Vibes (full app creation - food delivery) | App creation | 92% | 0/5 | All runs failed validation |
| New promotions submodule and active offers screen in Food Vibes (full app creation - food delivery) | App creation | 100% | 5/5 | N/A |
| New restaurant listing feed in Food Vibes (full app creation - food delivery) | App creation | 73% | 0/5 | All runs failed validation |
| New ViewModel layer in Fossify File Manager | New feature | 94% | 0/5 | All runs failed tests |
| Nav 2 to Nav 3 library migration in Bitwarden (password manager) | Migration | 93% | 0/5 | All runs failed tests |
| Dual-pane adaptive layout migration in Bitwarden (password manager): settings screens | Migration | 100% | 5/5 | N/A |
| Hilt to Koin dependency injection migration in Bitwarden (password manager) | Migration | 79% | 1/5 | 4 runs failed tests |
| XML views to Compose migration in Fossify File Manager: main tabs and settings | Migration | 74% | 0/5 | All runs failed tests |
| Hilt to Koin dependency injection migration in Now in Android (news app) | Migration | 47% | 0/5 | All runs failed tests |
| Kotlin Multiplatform migration in Now in Android (news app): datastore and database | Migration | 80% | 0/5 | All runs failed tests |
| Retrofit to Ktor migration in Pocket Casts (podcast player) | Migration | 100% | 5/5 | N/A |
| XML views to Compose migration in Signal (messaging app): custom chat colour creator | Migration | 80% | 0/5 | All runs failed tests |
| XML views to Compose migration in Signal (messaging app): profile creation | Migration | 97% | 0/5 | All runs failed tests |
| XML views to Compose migration in Signal (messaging app): phone number entry | Migration | 11% | 0/5 | All runs failed tests |
| Java to Kotlin migration in Signal (messaging app): Glide image loading module | Migration | 100% | 5/5 | N/A |
| XML views to Compose migration in Signal (messaging app): edit profile screens | Migration | 93% | 0/5 | All runs failed tests |
| XML views to Compose migration in WordPress: people invite screen | Migration | 35% | 0/5 | All runs failed tests |
| New book searching app using the OpenLibrary API | App creation | 100% | 5/5 | N/A |
Claude Fable 5 1
•
claude-code
Closed weights
model
Pass rate
22.7%
Avg completion rate
82.4%
Avg latency
22.2 h
Avg cost
$492.6
Per task results
| Task | Category | Avg completion rate | Runs passed | Failure reason(s) |
|---|---|---|---|---|
| Convert Flutter to Android: Habo app (habit tracker) | App conversion | 99% | 0/5 | All runs failed tests |
| Convert React Native to Android: BlueWallet app (Bitcoin wallet) | App conversion | 73% | 0/5 | All runs failed tests |
| New accessibility support across Droidcon (conference app) | New feature | 100% | 5/5 | N/A |
| New Media3 and ExoPlayer video playback in Now in Android (news app) | New feature | 93% | 2/5 | 3 runs failed tests |
| New picture-in-picture video mode in Now in Android (news app) | New feature | 94% | 1/5 | 4 runs failed tests |
| New CameraX profile picture capture in Now in Android (news app) | New feature | 80% | 4/5 | 1 run missed required test results |
| New user account submodule in Food Vibes (full app creation - food delivery) | App creation | 87% | 1/5 | 2 runs failed tests, 2 runs failed validation |
| New authentication flow with SMS verification in Food Vibes (full app creation - food delivery) | App creation | 60% | 0/5 | All runs failed validation |
| New bottom navigation and Browse search screen in Food Vibes (full app creation - food delivery) | App creation | 94% | 0/5 | 3 runs failed validation, 2 runs failed tests |
| New design system and reusable UI component library in Food Vibes (full app creation - food delivery) | App creation | 78% | 0/5 | All runs failed validation |
| New All Filters screen in Food Vibes (full app creation - food delivery) | App creation | 94% | 1/5 | 4 runs failed validation |
| New map integration and location features in Food Vibes (full app creation - food delivery) | App creation | 86% | 0/5 | All runs failed validation |
| New real-time order tracking in Food Vibes (full app creation - food delivery) | App creation | 90% | 0/5 | All runs failed validation |
| New promotions submodule and active offers screen in Food Vibes (full app creation - food delivery) | App creation | 100% | 5/5 | N/A |
| New restaurant listing feed in Food Vibes (full app creation - food delivery) | App creation | 71% | 0/5 | All runs failed validation |
| New ViewModel layer in Fossify File Manager | New feature | 94% | 0/5 | All runs failed tests |
| Nav 2 to Nav 3 library migration in Bitwarden (password manager) | Migration | 94% | 0/5 | All runs failed tests |
| Dual-pane adaptive layout migration in Bitwarden (password manager): settings screens | Migration | 80% | 4/5 | 1 run failed the build |
| Hilt to Koin dependency injection migration in Bitwarden (password manager) | Migration | 92% | 3/5 | 2 runs failed tests |
| XML views to Compose migration in Fossify File Manager: main tabs and settings | Migration | 73% | 0/5 | All runs failed tests |
| Hilt to Koin dependency injection migration in Now in Android (news app) | Migration | 56% | 0/5 | All runs failed tests |
| Kotlin Multiplatform migration in Now in Android (news app): datastore and database | Migration | 86% | 0/5 |