Official paper · code · released model
One image. Many questions. Shared compute.
Visual Jev scores answer choices from one image and shared context. Ask independent questions together; the model reuses the common prefix and batches the branches.
One shared input, isolated decisions
Visual Jev extends the Jev decision interface to vision. Each request supplies an image, optional shared text, and independent questions. The image and public prefix are encoded once; the question branches run together and cannot read each other's answers.
The image and common text become a cached prefix.
Each question sees its own instruction and choices, then runs in the same batch.
Candidate-token scores become a probability distribution; no new linear head is needed.
The paper evaluates choice decisions and three-way claim judgments. The released quickstart currently handles choice questions.
Fine-tuning helps. A new head does not.
Post-training for this interface lifts the equal-weight macro average over four benchmarks from 0.706 to 0.761. The gain is concentrated on the two families in the training mix; on the two held-out tasks it is small.
What does not show up is any contribution from the decision head. Answer SFT — same items, same steps, same prompts, same readout position, ordinary next-token cross entropy through the LM head — reaches 0.761. Three seeds each, ranges overlapping completely.
| System | Readout | Seeds | GQA | SNLI-VE | TextVQA† | TallyQA† | Macro |
|---|
† never seen in post-training. Macro weights the four benchmarks equally so the largest cannot carry it.
The gain appears when questions share an image
At 32 questions, shared-prefix batching takes 5.7 ms per question on an amortized basis, versus 50.7 ms for independent serial runs. Both batching and prefix reuse contribute to the 8.9× difference.
What this replaces
Today you would ask a vision–language model these questions by prompting it and reading what it writes — either one request per question, or all of them in one prompt and let it answer in a list. We measured both, on the same GPU, same model, same images and same questions.
Two honest caveats. The gap is a property of this workload — at N=1 there is no shared context to amortise and the ordinary way wins. And putting every question in one prompt lets them see each other, which is a different task: the answers can condition on one another, for better or worse.
On quality the comparable baseline is an off-the-shelf backbone asked to pick among the same candidates: 0.706 macro against 0.761 after post-training. We did not score the generative paths for accuracy — parsing free text into a choice is a different measurement, and reporting it beside ours would flatter us for their formatting errors.
Latency explorer
Warm per-question time from the sweep in the paper. Synchronized wall-clock timing from an in-memory decoded image and extracted records through path return; image decode, record construction and queueing are excluded. Two warm-ups are discarded on one RTX 5090, bfloat16, with 196 visual tokens.
Try the released 4B adapter
Use the included bottle image and six-question request. The runner downloads the Qwen3-VL-4B backbone and answer-supervised LoRA adapter on first use. It supports Apple Silicon MPS and CUDA.
After installing the dependencies in the quickstart guide, run from the repository root:
python code/examples/quickstart.py \
--image assets/figures/demo_bottle.jpg \
--request-file code/examples/bottle_questions.json --device autoIt returns a choice and probabilities for each question. Choices come from the request; a missing correct choice is not invented by the model. See the paper reproduction guide for training and evaluation.
One image, six questions
Held-out GQA images with the questions that naturally occur on them, and the probabilities the model actually returned. Each question is answered independently — it never sees the others — which is what lets all six share one visual encoding. The gold answer is in bold.
Confident without the evidence
Each row is one judgement under three conditions: the original image, the same image with the region the question depends on destroyed, and a control with an equal-area region elsewhere destroyed. All three were chosen because the model answers correctly with the evidence and incorrectly without it.
Watch the two signals diverge. Confidence barely moves even where the answer becomes wrong — it is still above 0.6 on every one of these. The sufficiency output, trained on these pairs, collapses instead.
It is not a clean instrument, and the rows show that rather than hide it: on the first, sufficiency also drops on the control, where the question's evidence is untouched. The paper measures how often that happens — the drop on the relevant arm is about five times the drop on the control, not infinitely more.
The incorrect_question choice in the demo can catch a false premise. This intervention asks a different question: does the image still contain the evidence needed for an otherwise valid answer?
What didn't work
Four things we built or expected that the measurements did not support. They are here because each one changes what a reader should do next.
The decision head
Against answer SFT at matched data and budget, on four benchmarks and three seeds. The interface is worth building; the head on top of it is not what makes it work.
Evidence-sufficiency objectives
The output detects removed evidence far better than confidence does — and still costs macro accuracy. Knowing evidence is missing is not knowing an answer is wrong.
Twice the parameters
8B against 4B, same recipe and GPU, for +28% latency and +80% memory. Barely outside seed noise, and an order of magnitude less than the serving path.
A trap, not a result
Train a slot-indexed head only on K≤4 and slots 4+ never get a gradient. The held-out benchmark then reads as a collapse in generalisation. It isn't one — it is the data.
Execution paths at N=32
| Path | Reuses | Batches | ms / question | Questions / s | Peak GiB |
|---|
Accuracy is identical across every readout path; the residual numerical difference is bfloat16 arithmetic, and vanishes in float32.