For years, detecting a specific object in a web app meant one thing: a bottleneck. Say you wanted to find 18th-century woodcut illustrations, or the diode on a vintage camera. You gathered thousands of images, labeled them by hand, and trained a custom model. Plenty of those projects burned through the budget before anyone wrote a line of production code.
So most of us skipped the feature, or settled for a rigid pre-trained model. Gemini object detection works differently. It does open-vocabulary spatial understanding, which means you describe the thing you are after in plain English instead of teaching a model to recognize it first.
The traditional ML mistake vs. spatial understanding
In the “old days,” which here means last year, you reached for YOLO or SSD. Those are fine on cats and cars. They fall apart on unstructured or distorted input: scanned archives and messy product photos throw them off with perspective shifts and paper grain.
Gemini 2.5 and Gemini 3 handle it through spatial understanding. The model reads the image against your prompt rather than matching a fixed mathematical pattern, so page curvature, tilted visuals and objects half hidden behind a bookmark or a stain still get picked up.
Structured output for detection
A text response is not enough for production. Pair the Google Gen AI SDK with Pydantic and you can pin the reply to a schema that hands back normalized bounding boxes, which is what an automated cropping or restoration pipeline actually needs to work with.
import pydantic
from google import genai
from google.genai.types import GenerateContentConfig
# Define the object structure for the API
class DetectedObject(pydantic.BaseModel):
box_2d: list[int]
label: str
caption: str
# Config for deterministic JSON output
config = GenerateContentConfig(
temperature=0.0,
response_mime_type="application/json",
response_schema=list[DetectedObject],
)
# The prompt is the "Training"
prompt = "Detect every illustration and extract bounding boxes and captions."
# response = client.models.generate_content(contents=[image, prompt], config=config)Beyond detection: editing with Nano Banana
With coordinates in hand from the Gemini object detection pass, the editing step is where it gets interesting. Google’s “Nano Banana” models (Gemini 2.5 Flash Image) do restoration work, and a descriptive prompt can push a detected region into a completely different style, watercolor or a “cinematized” movie still.
Legacy site migrations are where I have gotten the most out of this. Say you inherit 10,000 scanned pages of old manuals. A script can find the diagrams, straighten them, strip the yellowing and drop them into a modern WordPress layout as clean SVG or PNG assets. Eighteen months ago a solo dev or a small agency had no realistic way to do that.
Related reading: my notes on Agentic AI for repositories, and how we are experimenting with AI in WordPress site building.
Performance and production gotchas
Do not push this straight onto a production server and hope. Race conditions and token limits both bite. High-resolution images in particular will hit rate limits fast unless you wrap the calls in retry logic (the tenacity library does the job) or hand the work off to a task runner.
The constraints worth knowing before you start:
- Media resolution: tiny components such as circuit board labels need
MEDIA_RESOLUTION_ULTRA_HIGH. It costs more tokens and it stops the model from guessing. - Rate limiting: handle the 429s. The free tier of Google AI Studio throttles you quickly.
- Spatial normalization: boxes come back on a 0-1000 scale, so map them onto the real pixel dimensions of your source image before you crop.
If Gemini object detection is eating your dev hours, I can take it off your plate. I have been wrestling with WordPress and API integrations since the 4.x days.
Open-vocabulary detection changes the job
Going from fixed-label ML to open-vocabulary AI feels about as big as the jump from static HTML to the REST API. The work changes shape too. Less training models, more designing the prompts and pipelines that connect a model to real business data. Google’s Gemini Image Understanding documentation has the official details.
