Same messages, four apps
The same 20 everyday messages went to each app, one fresh chat per message. A judge model scored every reply without seeing which app wrote it, in shuffled order.
| App | Quality | Realism | Judged | Repeated openings | Median length | Median reply time | How it was sampled |
|---|
Persona variety and memory honesty (Soulor only)
Variety here means personas: the same messages go to six Soulor companions with different personalities, and the score is how different their replies to the same message are from each other. Personas should not sound like one voice in different hats. The other apps would need several characters sampled by hand, so this is Soulor's own number, not a comparison.
Anti-hallucination: three long chats plant facts among filler turns. At the end, each question is asked on its own: recall a planted fact; answer about something the person never said (inventing an answer fails); and, in a fresh chat with no history, "do you remember…" (pretending to remember fails).
Timed run
Fixed situations sent one at a time to Soulor's production AI gateway, using the real persona prompts. Time is the full request, from sending to the reply arriving.
Loading…
Production replies
Every reply Soulor serves is logged with the time it took. These are those numbers, per feature. Small samples are shown as they are.
| Feature | App | Replies | Median | 90th percentile |
|---|---|---|---|---|
| Loading… | ||||
What this does and doesn't measure
Quality and realism are one judge model's opinion, scored 1 to 10. The 21 October run's judge used the same hosted providers as Soulor, so it may lean towards Soulor's style. From the next run on, the judge must come from a different provider from the one that wrote Soulor's replies, and both are named under the table. Either way it is one model's taste; read those two bars with that in mind. A message whose judgement failed twice is left out, and "Judged" says how many were scored.
Word variety inside one character's 20 replies is no longer scored: it mostly tracks reply length, and it says nothing about whether different personas sound different. Persona variety is measured above, on Soulor only. Repeated openings still counts replies that start with the same two words as an earlier one, a sign of canned replies.
The memory questions are scored by expected words (recall) and by a judge model from a different provider from the one that wrote the replies (never said, fresh chat); the judge is named under the tiles.
Only Soulor's replies were timed. The other apps were sampled by hand in their own apps, so their reply time is "not timed".
Apps without a public API were sampled by hand in their own apps, on the account and character named in the table. An app that hasn't been sampled yet shows as not measured, never as an estimate.
A Simulation run asks seven perspectives at once, so the whole panel arrives in about the time of its slowest perspective, not seven times one.
Companion chat isn't in the timed run, because running it can start Soulor's own GPU model, which the run shouldn't pay for. Its production numbers are in the table when there are any.
Failed requests write no log row, so the production table says nothing about error rates. The timed run counts failures.
The raw numbers
Two scripts produce this page: one reads the production log, the other does the timed run. Their output is /data/benchmark.json, the file this page reads, so every figure here can be checked against it.