6 Vision API Models Tested: Gemini 2.5, GPT-4.1, and Qwen3 VL for Image Understanding
6 Vision API Models Tested: Gemini 2.5, GPT-4.1, and Qwen3 VL for Image Understanding
If you are building image understanding into a product, the phrase “supports images” is not enough.
A model page can say that a model supports vision input, but your production route still has to answer harder questions:
Does the OpenAI-compatible image_url payload actually reach the model?
Does HTTP 200 mean the model really saw the image?
Which route is fast enough for user-facing image uploads?
Which route is cheap enough for bulk image classification?
Which route should be used as fallback when the first model fails?
Can usage metadata reveal a broken media path?
To make the comparison more practical, I tested six vision-capable models through the same OpenAI-compatible API shape:
The goal is not to crown a universal winner. The useful question is: which model should you route to for a specific user workflow?
All tests used the Crazyrouter OpenAI-compatible Base URL:
https://cn.crazyrouter.com/v1
The request format was chat/completions, with the image passed through messages[].content[] as an image_url object.
Each model was tested on two stable public images:
Each image was run three times per model, so every model had six requests in total.
Test time: 2026-06-21T13:36:32Z.
This is a vision API smoke test. It is useful for checking whether the image_url route works and whether the model can perform simple visual recognition. It is not a full OCR, chart reasoning, document extraction, handwriting, or medical-image benchmark.
Real-time user uploads / lowest latency: use gpt-4.1-mini
Bulk logo, icon, or simple image classification: use qwen3-vl-flash
Low-cost Gemini route: consider gemini-2.5-flash-lite
Low-cost OpenAI-family route: consider gpt-4.1-nano
Quality-oriented Qwen VL route: use qwen3-vl-plus as an upgrade path
Do not use as default image_url vision route in this run: gemini-2.5-flash
The most important finding is simple:
HTTP 200 does not prove image understanding succeeded.
In this test, gemini-2.5-flash returned HTTP success for all six requests, but the visual recognition score was 0/6. It also produced outputs such as “no image provided,” incorrect CBC logo recognition, and unrelated object descriptions.
That is the dangerous failure mode: the API call appears successful, but the model did not correctly process the image.
ModelHTTP successCorrect recognitionNo-image repliesAvg latencyMedian latencySlowest requestInput price / 1M tokensOutput price / 1M tokensEstimated cost / 10k test-style callsPositioningqwen3-vl-flash6/66/603.819s3.493s5.975s$0.05$0.40$0.0915Best low-cost route for bulk recognitiongpt-4.1-mini6/66/601.491s1.292s2.189s$0.26$1.04$0.5226Best low-latency route for user-facing featuresgpt-4.1-nano6/66/602.863s2.562s4.213s$0.065$0.26$0.1666Low-cost OpenAI-family routeqwen3-vl-plus6/66/603.859s3.729s4.821s$0.1429$1.4286$0.3848Quality-oriented Qwen VL upgrade routegemini-2.5-flash6/60/614.965s4.333s9.507s$0.17$0.68$0.6168image_url path failed in this rungemini-2.5-flash-lite6/66/602.618s2.627s4.195s$0.055$0.22$0.5466Low-cost Gemini lightweight route
The estimated 10k-call cost is based on observed usage in this simple logo recognition test. It should not be treated as universal pricing for all image workloads. Larger images, OCR tasks, long descriptions, and multi-image prompts can change token usage significantly.
The useful production metric is not just model price. It is cost per successful image task.
A cheap route that frequently needs retries or fallback can be more expensive than a more reliable route.
Accuracy: five models passed, one model failed
Correct recognition in this smoke test:
gemini-2.5-flash-lite: 6/6
For simple logo and icon recognition, five of the six routes worked correctly. This means many lightweight models are already enough for basic image classification.
But gemini-2.5-flash is the important cautionary example: HTTP success does not mean the image path is healthy.
Latency: GPT-4.1 Mini was the fastest
Average latency from lowest to highest:
gpt-4.1-mini: avg 1.491s, median 1.292s, slowest 2.189s
gemini-2.5-flash-lite: avg 2.618s, median 2.627s, slowest 4.195s
gpt-4.1-nano: avg 2.863s, median 2.562s, slowest 4.213s
qwen3-vl-flash: avg 3.819s, median 3.493s, slowest 5.975s
qwen3-vl-plus: avg 3.859s, median 3.729s, slowest 4.821s
gemini-2.5-flash: avg 4.965s, median 4.333s, slowest 9.507s
For user-facing features, latency is part of product quality. If the user uploads an image and waits for a response, one or two seconds can matter.
For those workflows, gpt-4.1-mini is the strongest default route in this run.