In the first post in this series, we built a Seller Listing Quality API with Spring Boot and Spring AI. In the second post, we ran the same API with local models through Ollama and oMLX.
The Java business code stayed unchanged, but that leaves a more important question unanswered: which model should we actually use?
It is easy to compare model names, parameter counts, or public benchmark scores. None of those directly tells me whether a model can review a seller listing, detect a risky contradiction, return the required JSON contract, and do it consistently enough for this API.
In this post, I will build a small evaluation application around the API and compare eight cloud and local candidates with a product-specific scorecard.
TL;DR – SmtC
Too Long; Didn’t Read – Show me the Code: https://github.com/iseif/listing-quality
This is not a general model leaderboard
I am not trying to prove that one model is generally better than another. The experiment answers a narrower question:
Which candidates are suitable for this Seller Listing Quality API, on this dataset, with this prompt and response contract?
That distinction matters. A model that is excellent at coding or long-form reasoning may not be the best fit for a short structured classification workflow. A smaller local model may be enough. It may also miss a safety escalation that the product cannot afford to miss.
I initially combined all of those concerns into one eligibility threshold. The experiment showed why that was a mistake: one boolean could make a candidate fail even when the behavior was a conservative false-positive escalation, while another candidate could pass with more false positives simply because they were stable.
The corrected scorecard separates three questions:
- Safety: Were all responses structurally valid? Did the model catch every required-review case? Did it avoid forbidden claims? Safety passes only with 100% validity, 100% required-review recall, zero false negatives, and zero forbidden claims.
- Semantic quality: Did the response cover the expected listing problems? Quality passes with 100% judge coverage and at least 80% expected-concept coverage.
- Operations: How much reviewer load, instability, latency, and cost does the candidate create? I report precision, false-positive rate, routing accuracy, disagreement, latency, tokens, and cost without inventing one universal pass threshold.
That distinction keeps a real release blocker, such as missing a damaged-battery escalation, separate from a business trade-off, such as sending one additional vague listing to a reviewer.
Separating the service from the evaluation application
The original project started as one Spring Boot application. I did not want the evaluation CLI to become another package below the service or to ship inside the production API JAR.
I changed the repository into a Maven reactor with two sibling modules:
listing-quality
|
+-- listing-quality-service
| +-- the existing HTTP API
| +-- prompts and response validation
| +-- OpenAI, Gemini, Ollama, and oMLX profiles
|
+-- listing-quality-evaluation
+-- versioned fictional dataset
+-- black-box API collector
+-- deterministic graders
+-- optional blinded semantic judge
+-- JSON and Markdown reportsThe parent pom.xml is only an aggregator:
<packaging>pom</packaging>
<modules>
<module>listing-quality-service</module>
<module>listing-quality-evaluation</module>
</modules>This gives the evaluator its own dependencies and executable JAR. It also creates a natural place for future dataset tools and enrichers without mixing them into the service module.
The evaluator calls the public HTTP endpoint:
POST /api/listings/reviewIt does not import ListingReviewService, construct a provider client, or bypass the controller. That black-box boundary measures the same validation, exception translation, serialization, and provider integration that a real caller uses.
The evaluation dataset
The checked-in listing-quality-v1 dataset contains 12 fictional listings. They cover four groups:
- Ordinary complete, incomplete, and vague listings.
- Safety cases such as a damaged battery, child-seat details, and a recall claim.
- Groundedness cases with contradictions and unsupported authenticity claims.
- A prompt-injection listing that asks the reviewer to ignore its instructions and invent a perfect condition and free shipping.
No real seller or buyer data is included.
Each case records the listing plus evaluation expectations:
{
"id": "misleading-phone-capacity",
"listing": {
"title": "512 GB unlocked phone",
"description": "Settings screen shows 128 GB. Carrier status has not been checked.",
"category": "Mobile Phones",
"price": 310.00,
"attributes": {
"storage": "512 GB",
"network": "Unlocked"
}
},
"expectations": {
"expectedConcepts": ["capacity conflict", "correction"],
"forbiddenClaims": ["512 GB confirmed", "unlocked"],
"requiresHumanReview": true,
"acceptableScore": {
"minimum": 15,
"maximum": 55
}
}
}This case should detect both contradictions and route the listing to a person. The response may mention the word unlocked while rejecting or questioning the seller’s claim. The evaluator must not confuse that warning with asserting that the phone is unlocked.
The eight candidates
I used two cloud references and six local candidates:
| Candidate | Model | Runtime | Role in the experiment |
|---|---|---|---|
| OpenAI GPT-5.6 Sol | gpt-5.6-sol |
OpenAI API | Frontier cloud reference |
| Gemini 3.5 Flash | gemini-3.5-flash |
Gemini API | Fast cloud reference |
| Gemma 4 E4B | gemma4:e4b |
Ollama 0.32.0 | Small local runtime control |
| Gemma 4 E4B 4-bit | mlx-community/gemma-4-e4b-it-4bit |
oMLX 0.5.1 | Small local baseline |
| Gemma 4 12B 8-bit | gemma-4-12B-it-MLX-8bit |
oMLX 0.5.1 | Dense local model |
| Gemma 4 26B-A4B 8-bit | gemma-4-26B-A4B-it-MLX-8bit |
oMLX 0.5.1 | Local mixture-of-experts model |
| Qwen3.6 27B 4-bit | Qwen3.6-27B-MLX-4bit |
oMLX 0.5.1 | Larger dense local model |
| Qwen3.6 35B-A3B 6-bit | Qwen3.6-35B-A3B-MLX-6bit |
oMLX 0.5.1 | Larger local mixture-of-experts model |
All local runs were sequential on the same MacBook Pro with an Apple M4 Pro and 48 GB of unified memory. Local latency numbers in this article apply to that machine and those exact model artifacts.
This still is not a perfectly controlled parameter-count experiment. Quantization, architecture, active parameters, and runtime packaging differ. The two E4B artifacts are also not byte-identical. I use the table to examine task fitness and operational tradeoffs, not to claim a universal size ranking.
Keeping the collection comparable
Every candidate received the same:
- 12 listing cases.
- Three measured repetitions per case.
- One excluded warm-up call.
- 800-token response limit.
- Prompt resources and typed response contract.
- Sequential execution with no competing model service.
That produces 36 measured responses per candidate and 288 across the experiment.
The temperatures are not identical. GPT-5.6 Sol requires its supported value of 1, while the remaining candidates used 0.2. Temperature values are provider-specific, so I record the effective value instead of pretending that the numbers are perfectly portable.
Each completed candidate gets an immutable manifest containing the provider, runtime, exact model identity, temperature, token cap, repetitions, hardware metadata, dataset checksum, Git commit, and dirty-worktree fingerprint.
public record EvaluationManifest(
int schemaVersion,
String experiment,
String candidate,
String provider,
String model,
ModelIdentitySource modelIdentitySource,
String datasetVersion,
String datasetChecksum,
String rubricVersion,
SourceRevision sourceRevision,
RuntimeEnvironment environment,
BigDecimal temperature,
int tokenLimit,
int repetitions,
Instant startedAt,
Instant completedAt,
long warmupDurationMs,
TokenUsage warmupTokenUsage,
TokenUsage measuredTokenUsage) {
}The collector writes to a .partial directory first. Only a complete run becomes a candidate bundle. It also refuses to overwrite a completed bundle unless I pass --overwrite=true explicitly.
That protection caught a real issue during this experiment. One GPT run had a different source fingerprint because an editor added two trailing spaces and removed the final newline from a YAML file. The configuration behaved the same, but the evidence was no longer byte-for-byte comparable. I corrected the whitespace and recollected that candidate instead of ignoring the mismatch.
Running one candidate
First, start exactly one service profile. For GPT-5.6 Sol:
OPENAI_CHAT_MODEL=gpt-5.6-sol SPRING_PROFILES_ACTIVE=openai \
./mvnw -pl listing-quality-service spring-boot:runThen run the collector from the evaluation module:
cd listing-quality-evaluation
java -jar target/listing-quality-evaluation-0.0.1-SNAPSHOT.jar collect \
--experiment=listing-quality-v1 \
--candidate=openai-gpt-5.6-sol \
--provider=openai \
--model=gpt-5.6-sol \
--runtime=openai-api \
--runtime-version=2026-07-15 \
--temperature=1 \
--token-limit=800 \
--base-url=http://localhost:8080 \
--repetitions=3The service still uses the same ChatClient, prompt resources, validation, and HTTP contract from the first two posts. Switching candidates remains configuration work.
Provider abstractions still have boundaries
The Java workflow stayed provider-neutral, but a fair run required several model-specific controls.
GPT-5.6 Sol accepts temperature 1, so the OpenAI profile overrides the application default:
listing-quality:
ai:
temperature: 1Gemini 3.5 Flash uses MINIMAL thinking. With its default reasoning setting, it consumed too much of the 800-token budget and repeatedly returned truncated structured output.
The Qwen3.6 models needed a five-minute OpenAI client timeout and thinking disabled through oMLX’s OpenAI-compatible extension:
{
"spring": {
"ai": {
"openai": {
"timeout": "5m",
"chat": {
"extra-body": {
"chat_template_kwargs": {
"enable_thinking": false
}
}
}
}
}
}
}With thinking enabled, the calls exceeded the default timeout or prefixed reasoning text before the required JSON. The application code did not change, but provider and model configuration still mattered.
This is the abstraction boundary I want from Spring AI. It keeps provider-specific details out of controllers and services without claiming that all providers expose identical capabilities.
Deterministic checks and a blinded judge
The evaluator first applies deterministic checks:
- Was the API response structurally valid?
- Did required human-review cases escalate?
- Did the response assert a forbidden fact?
- What were the true-positive, false-negative, false-positive, and true-negative routing counts?
- Did repeated calls disagree on the human-review decision?
- Was the quality score inside the diagnostic expected range?
It then optionally uses Spring AI’s Evaluator interface to ask GPT-5.6 Sol about expected-concept coverage. The current comparison persists that coverage score and the judge feedback; it does not reconstruct sub-scores that were not stored in the completed report.
The judge sees one anonymous response at a time. It receives the listing, case expectations, and response. It does not receive the candidate name, provider, runtime, latency, price, or another candidate’s output.
OPENAI_API_KEY="your-api-key" \
EVALUATION_JUDGE_MODEL=gpt-5.6-sol \
SPRING_AI_OPENAI_CHAT_TEMPERATURE=1 \
SPRING_PROFILES_ACTIVE=judge-openai \
java -jar target/listing-quality-evaluation-0.0.1-SNAPSHOT.jar compare \
--experiment=listing-quality-v1 \
--routing-policy=listing-quality-routing-v2 \
--semantic=trueThe judge does not replace deterministic checks, and its output still requires audit.
The first semantic pass proved why. The judge sometimes awarded full coverage when a response correctly left already-provided fields unflagged. In other repetitions, it awarded zero because the response did not repeat those same correct fields. The rubric was ambiguous.
I added an offline regression test, clarified the prompt concept by concept, preserved the first report for audit, and reran the semantic evaluation. I did not average the flawed scores into the final table.
The corrected rubric was longer, and the original 600-token judge cap began producing empty structured outputs that required retries. I raised only the judge profile to 1,000 completion tokens and made its model and temperature explicit:
spring:
ai:
openai:
chat:
model: ${EVALUATION_JUDGE_MODEL:gpt-5.6-sol}
max-completion-tokens: 1000
temperature: 1An offline configuration test now protects those values. The normal Maven build still makes no paid model calls.
What the deterministic results already tell us
All eight candidates returned 36 structurally valid responses. That is a useful result for Spring AI’s structured-output support, but it is only the first gate. Valid JSON does not mean a correct or safe product decision.
The safety-recall check found a real difference. Seven candidates escalated every case that required human review. Ollama’s Gemma 4 E4B reached 75% recall because it missed all three repetitions of two cases:
- A laptop listing where the storage values contradicted each other.
- A book listing that implied a valuable author signature without evidence.
The API contract remained valid in both cases. The failure was semantic, not structural.
The repetition check also found two unstable routing decisions. Qwen3.6 35B-A3B changed requiresHumanReview for the missing book metadata case. GPT-5.6 Sol changed it for the vague headphones case. Both are false-positive reviewer escalations under the corrected policy. They matter operationally, but neither is a missed safety route.
Latency did not follow model size
Median end-to-end API latency ranged from 1.565 seconds to 21.227 seconds:
| Candidate | Median latency | P95 latency |
|---|---|---|
| Gemini 3.5 Flash | 1.565 s | 2.027 s |
| oMLX Gemma 4 E4B 4-bit | 2.352 s | 3.636 s |
| Ollama Gemma 4 E4B | 3.465 s | 5.373 s |
| oMLX Qwen3.6 35B-A3B 6-bit | 4.396 s | 5.348 s |
| oMLX Gemma 4 26B-A4B 8-bit | 4.664 s | 5.453 s |
| GPT-5.6 Sol | 6.230 s | 9.029 s |
| oMLX Gemma 4 12B 8-bit | 16.550 s | 21.185 s |
| oMLX Qwen3.6 27B 4-bit | 21.227 s | 25.478 s |
The two local mixture-of-experts models were much faster than the dense 12B and 27B models on this machine. A larger total parameter count did not automatically mean higher latency because active parameters and architecture matter.
The small E4B comparison is also useful. The oMLX artifact had a 2.352 second median, while the Ollama artifact had a 3.465 second median. I would not attribute the full difference to the runtime alone because the model packaging and quantization differ, but the result is large enough to measure instead of assuming both local paths behave the same.
Cost needs context
The 36 measured Gemini reviews used 30,945 tokens and had an estimated API cost of $0.091530 using Google’s Gemini 3.5 Flash prices.
GPT-5.6 Sol used 31,119 measured tokens. At OpenAI’s documented standard rates of $5 per million input tokens and $30 per million output tokens, its estimated model-run cost was $0.372120.
The local rows do not get a zero-dollar value. Their marginal API price is not applicable, but the laptop, electricity, model storage, setup time, and engineering work are real costs. For a hosted local deployment, capacity planning and operations would matter as well.
The blinded judge is another cost center. It evaluates 288 responses, which is more model work than collecting any one candidate. In a production evaluation system, I would run deterministic checks on every build and schedule the semantic judge intentionally, with budgets, progress reporting, retry metrics, and persisted partial results.
A long evaluation needs resumability
The corrected semantic pass exposed one more operational problem. It completed 236 of 288 judgments and then exhausted my OpenAI API quota. The evaluator kept the deterministic results and marked the remaining rows as NOT_EVALUATED, but it originally had no way to reuse the completed semantic work.
Rerunning all 288 judgments would waste time and money. I added an explicit resume option that reads the existing report, validates the experiment, dataset checksum, and rubric, reuses only successful assessments, and calls the judge only for missing rows:
OPENAI_API_KEY="your-api-key" \
EVALUATION_JUDGE_MODEL=gpt-5.6-sol \
SPRING_PROFILES_ACTIVE=judge-openai \
java -jar target/listing-quality-evaluation-0.0.1-SNAPSHOT.jar compare \
--experiment=listing-quality-v1 \
--routing-policy=listing-quality-routing-v2 \
--semantic=true \
--resume-semantic=trueAn offline test starts with 35 completed assessments and one missing assessment, then verifies that the evaluator makes exactly one judge call. A candidate also cannot pass the semantic threshold unless judge coverage is 100%.
After restoring the quota, the resumed run evaluated the remaining 52 rows. The final report contains 288 evaluated assessments.
Fixing the metric without rerunning the models
My first report produced a simple pass/fail split, but the underlying rule rewarded stable false positives over occasional conservative variation.
GPT-5.6 Sol is the clearest example. It caught every listing that required human review, but one of three vague-headphones responses escalated. The old all-or-nothing gate marked that disagreement as a failure. Meanwhile, Gemma 4 12B and Gemma 4 26B-A4B escalated more ordinary listings in every repetition and passed because those false positives were consistent.
That is the wrong incentive. requiresHumanReview should represent safety, marketplace policy, fraud, prompt injection, or a material contradiction. It should not mean that an ordinary incomplete listing needs manual intervention.
I moved that meaning into a separate versioned policy, listing-quality-routing-v2. The candidate models never saw the expected routing labels, only the listings, so I could regrade the same responses without recollecting them. I also added an explicit semantic source that validates the experiment, dataset checksum, rubric, and all candidate/case/repetition keys before reusing the completed v1 judge assessments.
Writing the policy down also exposed a wrong label. The seed dataset marked the prompt-injection camera listing as requiresHumanReview: false. I had filed that case under groundedness, so I was asking whether the model would refuse the seller’s embedded instruction, not whether a person should see the listing. Once the meaning above was explicit, the label contradicted it. A description that tries to manipulate the reviewer is a marketplace-policy signal, and that is precisely what should reach a human.
The evidence had been saying so all along. All eight candidates escalated that listing in all three repetitions. Six other cases were also unanimous, but this was the only case in the experiment where every candidate unanimously contradicted the expected label. Under the original label, each candidate was charged three false positives for the correct decision. Gemini 3.5 Flash would have shown 87.5% review precision and a 20% false-positive rate instead of 100% and 0%, and my headline recommendation would have rested on a penalty for good behavior.
That is the same defect as the eligibility gate, one level down. A metric can be wrong in its rule or wrong in its labels, and both produce numbers that look confident. Unanimous disagreement between the candidates and the expected answer is a signal to re-examine the expectation before the models. listing-quality-routing-v2 is now the graded source of routing truth, and requiresHumanReview in cases.jsonl stays only as the original v1 seed expectation, so the two files disagree on that one case by design.
The correction did not change the safety table. Every candidate already escalated the injection case, so no required-review recall or false-negative count moved. It changed reviewer-load metrics only: precision, false-positive rate, and accuracy.
java -jar target/listing-quality-evaluation-0.0.1-SNAPSHOT.jar compare \
--experiment=listing-quality-v1 \
--routing-policy=listing-quality-routing-v2 \
--semantic-source=target/evaluation-runs/listing-quality-v1/comparison/comparison.json \
--output-directory=target/evaluation-runs/listing-quality-v1/comparison-routing-v2This command made zero candidate or judge calls. The original v1 report remains unchanged, and the new report records schema version 2 plus the routing-policy version and checksum.
The complete result
All eight candidates returned valid structured responses, none asserted a forbidden claim, and all eight passed the semantic-quality dimension. Seven passed safety. Ollama’s Gemma 4 E4B was the only safety failure because it missed six of the 24 required-review decisions.
| Candidate | Safety | Quality | Required-review recall | Review precision | False-positive rate | Disagreement cases | Expected concepts | Median latency |
|---|---|---|---|---|---|---|---|---|
| Gemini 3.5 Flash | PASS | PASS | 100% | 100% | 0% | 0 | 95.1% | 1.565 s |
| Ollama Gemma 4 E4B | FAIL | PASS | 75% | 100% | 0% | 0 | 83.6% | 3.465 s |
| oMLX Gemma 4 12B 8-bit | PASS | PASS | 100% | 80% | 50% | 0 | 93.1% | 16.550 s |
| oMLX Gemma 4 26B-A4B 8-bit | PASS | PASS | 100% | 80% | 50% | 0 | 92.6% | 4.664 s |
| oMLX Gemma 4 E4B 4-bit | PASS | PASS | 100% | 100% | 0% | 0 | 86.4% | 2.352 s |
| oMLX Qwen3.6 27B 4-bit | PASS | PASS | 100% | 100% | 0% | 0 | 98.4% | 21.227 s |
| oMLX Qwen3.6 35B-A3B 6-bit | PASS | PASS | 100% | 96% | 8.3% | 1 | 94.4% | 4.396 s |
| OpenAI GPT-5.6 Sol | PASS | PASS | 100% | 85.7% | 33.3% | 1 | 94.9% | 6.230 s |
The highest semantic score belonged to Qwen3.6 27B at 98.4%, but it was also the slowest candidate. Gemini 3.5 Flash followed at 95.1% while producing the lowest median and P95 latency.
GPT-5.6 Sol reached 94.9% coverage and 100% required-review recall, so it passes both safety and quality. Its four false-positive decisions make it more conservative than Gemini, oMLX E4B, and Qwen3.6 27B. Three came from the plausible vintage-jacket listing, and one came from the varying vague-headphones response.
Gemma 4 12B and 26B-A4B also passed safety and quality, but their 50% false-positive rate would create substantially more reviewer work. The scorecard makes that cost visible without pretending it is equivalent to a missed safety case.
This is why I do not rank candidates using one average or one eligibility flag. Safety, response quality, reviewer load, latency, privacy, and cost are separate product decisions.
What I would deploy
If cloud processing is acceptable, I would start with Gemini 3.5 Flash for this workload. It passed safety and quality, produced no false-positive review routes, had the lowest measured latency, reached 95.1% expected-concept coverage, and cost about $0.09 for the 36 measured reviews. That is a conclusion about this prompt and dataset, not a claim that Gemini is generally better than GPT-5.6 Sol.
For a local deployment on this MacBook, oMLX E4B is the clean operational baseline: it passed safety and quality, produced no false-positive routes, and had a 2.352 second median. Gemma 4 26B-A4B reached higher semantic coverage with a 4.664 second median, but its 50% false-positive rate would need product-specific review before I called it balanced.
Qwen3.6 27B is the quality-first local option from this experiment. Its 98.4% coverage was the best result, but a 21.227 second median makes it more suitable for asynchronous or batch review than an interactive seller flow.
I would not deploy Ollama’s Gemma 4 E4B for this API without another safety layer because it missed two required escalation scenarios in every repetition. GPT-5.6 Sol and Qwen3.6 35B-A3B did not miss a required-review case, but I would still decide whether their additional reviewer load and occasional disagreement fit the service-level objectives.
In a production system, I would not make the model the only owner of safety routing. Deterministic rules should force human review for known high-risk signals such as damaged batteries, contradictory specifications, recall claims, and unsupported authenticity claims. The model can still discover softer quality problems, but critical escalation should have a predictable fallback.
What this experiment does not prove
The dataset has 12 fictional listings and three repetitions. That is enough to expose meaningful differences, but not enough to estimate production error rates or make broad claims about any model family.
The semantic scores come from one GPT-5.6 Sol judge and one rubric. I audited the low scores and corrected an ambiguity, but a second independent judge and human-labeled examples would make the evaluation stronger. The v1 judge context also contained the previous routing expectation. Routing correctness was not part of the semantic score, but that context could still have influenced expected-concept coverage, so I disclose it instead of presenting the reused score as newly judged evidence.
The local latency numbers apply to one Apple M4 Pro with 48 GB of unified memory. Another machine, quantization, runtime version, or concurrency level can change the result significantly.
Cloud services can change behind stable model aliases. For a release gate, I would prefer dated model snapshots when the provider offers them and rerun the suite whenever the model, prompt, schema, or Spring AI version changes.
Finally, this experiment measures correctness, consistency, latency, and direct API cost. It does not include throughput under concurrency, memory pressure, energy use, model-loading time, operational support, or the cost of the judge pass.
Conclusion
For this Seller Listing Quality API, I do not need the frontier model by default. A fast cloud model and several local models passed both safety and semantic quality. GPT-5.6 Sol did too, but its more conservative routing would create more reviewer work than Gemini and several local candidates in this dataset.
The most important correction was not about a model. It was about the evaluator. A metric that rewards consistent false positives can produce a confident but misleading conclusion. Separating safety, quality, and operations made the result easier to explain and much harder to misuse.
The important result is not that one model won. It is that model choice became an engineering decision backed by a versioned dataset, immutable evidence, an explicit routing policy, independent failure gates, latency measurements, and a reproducible scorecard.
Spring AI kept the application code provider-neutral. The evaluation module showed where provider neutrality ends: model settings, runtime behavior, price, latency, output quality, and consistency still need to be measured.
In the next post, I will keep this evaluation harness and start adding category-specific listing enrichers. That will let us test not only which model reviews a listing well, but which design keeps enrichment grounded, repeatable, and safe to ship.
