Serving WeVisOne-VL-4B: two engines, two copies of the weights, one chess piece apart
Before you put WeVisOne-VL-4B behind an API, decide which engine writes the answer. The authors' package runs the model two ways: through vLLM, a server built for throughput, or through plain Hugging Face Transformers as the reference. That choice sets your memory, your latency and, on one test image, whether the count was right.
The model comes from the WeChat Vision team at Tencent, on top of Alibaba's Qwen3-VL-4B-Instruct. One 4B checkpoint does the work of a detector, a segmenter and a matting tool. It boxes what you name, counts, points, reads text with its position, outlines and cuts out. We deployed it on our own cloud and sent the same 14 requests to each setup. Tables are in the full WeVisOne-VL-4B test with every measurement.
Why the fast path holds the model twice
Language models write tokens, word pieces produced one at a time. This one also has position tokens: four make a box, two a point. For outlines and cut-outs it writes a marker token after each box. A separate cutter of 9.6 million parameters (mask_decoder.pt) then reads the model's internal state at that marker and paints the pixels.
vLLM hands back the text, not those internal states. So with backend="vllm" the package also loads the full Transformers model and replays the answer through it for the cutter. At half precision the weights weigh about 10 GB, and this path keeps two copies. On an RTX 4090 it peaked at 23.0 GiB of 24 GB, against 12.3 GiB for Transformers alone.
The default settings do not start on 24 GB
With the package's defaults on a 4090, vLLM refuses to boot. It reserves room for ten images and ten videos per prompt, and that leaves nothing for the KV cache, the scratch memory where a model keeps what it has already read and written:
Encoder cache will be initialized with a budget of 16384 tokens
Available KV cache memory: -0.87 GiB
ValueError: No available memory for the cache blocks
Our custom backend, the small Python program our platform runs on the GPU machine instead of a standard model server, wraps vllm.LLM with one image per prompt, no video, a shorter context (the most tokens one request may hold) and at most four requests in flight. Giving vLLM 0.5 of the card next to the second copy made the big portrait cut-out run out of memory. At 0.465 with a 4096-token context, the cache came up short. At 0.465 with 3072 tokens, all 14 requests passed. Thin margin: measure peak memory with your own image sizes.
Same greedy decoding, different answers
On the authors' chess photo (16 white pieces), vLLM said 15 in 1.35 seconds, missing the knight on g5. Transformers said 16 in 3.87 seconds.
!Side by side, WeVisOne-VL-4B counting white chess pieces with two engines on an RTX 4090. Left, vLLM: 15 in 1.35 seconds, the white knight on g5 marked as missed. Right, Transformers: 16 in 3.87 seconds, the knight boxed
Both runs used greedy decoding, meaning the model always takes its single most likely next token, so no dice are rolled. Most likely the two programs round their maths a little differently, and a box on the edge tipped over. They also split on a street sign (29 words against 21) and on hanging pots (19 against 20). One wrong count out of 14 says nothing about a rate. It does say: run your evaluation set on the engine you will ship.
What to measure before you ship
- The first request after a start. On vLLM, a detection that runs in 1.9 seconds once warm took 14.5 to 15.5 seconds the first time. Fire a throwaway request at boot.
- Peak memory per task, not per model. The cutter asks for about 0.8 GB at once, at any photo size. With the 4090 capped like a strict 12 GB card, the text jobs (boxes, counts, points, reading) still passed. Outlines and cut-outs all failed, even on a 640-pixel photo.
- Response size. The platform dropped results above 8 MB, and the caller waited for a 504, the HTTP gateway-timeout error. The backend now fits a 5.2 MB budget (JPEG overlay, 1280 px cut-out). The platform side, where the client waits out the full timeout, is open.
The reference engine is the safer default
The smallest card that passed all 14 requests was a real RTX 5080 with 16 GB, on Transformers only, peaking at 12.8 GiB. Against the 4090's fast engine it took roughly double the time on the big cut-out (2.1 seconds for a 1620 by 1080 portrait, against 1.0) and up to four times as long on object jobs.
The backend picks vLLM on cards with 22 GiB or more in total, Transformers below. Release 0.1.3 makes the tested values the defaults, but has not itself been run on a GPU. 0.1.0 to 0.1.2 were, with the values passed by hand. The 12 GB and 8 GB figures are caps applied on the 4090, not real smaller cards. We sent one request at a time, so concurrency is untested.
The authors claim the lead on over 30 benchmarks (55.3 against GroundingDINO's 38.8 on finding named objects). We did not rerun them; we judged outputs by eye and counted the chessboard by hand.
The WeVisOne-VL-4B deployment write-up on inferencekey.com has the timing table, the VRAM ladder down to an 8 GB cap with the 2B model, the environment variables and each problem's status. The code is in the SDK example for WeVisOne-VL-4B.
Credit: Tencent's WeChat Vision team built the model, the code package and the model card. Qwen3-VL underneath is Alibaba's. The vLLM and Transformers projects make both engines, and the AI Search channel's news video put the model on our radar. Our part was deploying it, measuring it and writing the backend. Read the complete WeVisOne-VL-4B results, tables and setup before you size your card.
How to Run Tencent WeVisOne-VL-4B Locally on 8 GB VRAM (Ollama Drops the Masks)
Tencent's WeVisOne-VL-4B picked up 5,986 Hugging Face downloads last month, and I suspect a lot of those people expected another small chat-about-your-image model. It's a perception model instead: boxes, points, counts, text locations, segmentation masks and alpha mattes, all from one 4B checkpoint built on Qwen3-VL. You can run WeVisOne-VL-4B locally on an 8 GB GPU with a Q4_K_M GGUF in llama.cpp, and the Hub offers a one-line Ollama pull too. You just lose the masks when you do.
I went through the model card, the GitHub README, the config and the GGUF repos. This post covers what the model is, Tencent's numbers against Qwen3-VL-8B (and the baseline they left out), VRAM per format, and the two ways to run it.
What WeVisOne-VL-4B actually is
The model card says it comes from WeChat Vision at Tencent and is built on Qwen3-VL. The config.json confirms it: Qwen3VLForConditionalGeneration, 36 text layers, hidden size 2560, a 24-layer ViT with patch size 16. That's the Qwen3-VL-4B layout with a 152,680-token vocabulary.
The extra tokens are the trick. Coordinates are emitted as <0> to <1000> (normalized boxes and points), and every predicted box in dense tasks is followed by an <OBJ> token (ID 152678). A separate 9.6M-parameter pixel decoder, shipped as mask_decoder.pt, reads those <OBJ> positions plus multi-level ViT features and produces masks and stride-1 alpha mattes, so there's no SAM in the loop.
The task list from the GitHub repo: detection, referring, point, count_box / count_point, ocr / ocr_grounding, segmentation, matting.
Other facts worth knowing before you download:
- License: Apache-2.0 for Tencent-authored code; third-party components keep their own licenses.
- Context: max_position_embeddings is 262144, same as the Qwen3-VL base. Tencent's evals cap images at 1,960 visual tokens, so context isn't the constraint here.
- Single-turn, single image. The pixel decoder rejects video grids, and the model returns structured output without chain-of-thought.
- It answers None when the thing you asked for isn't in the image, which matters more than it sounds (see the benchmarks).
- There's also a 2B sibling. Paper is arXiv:2610.31329, submitted 22 Oct 2026.
WeVisOne-VL-4B vs Qwen3-VL benchmarks, and who's missing
All numbers below are from the GitHub README. Tencent compares against Qwen3-VL-8B, Qwen3.5-9B, Rex-Omni-3B, LocateAnything-3B and a few specialists.
| Task (metric) | Qwen3-VL-8B | Qwen3.5-9B | Rex-Omni-3B | LocateAnything-3B | WeVisOne-VL-4B |
|---|---|---|---|---|---|
| LVIS detection (F1) | 42.9 | 43.7 | 46.9 | 50.7 | 55.3 |
| Dense200 detection (F1) | 13.8 | 18.0 | 58.3 | 58.7 | 71.3 |
| RefCOCOg test (F1) | 74.9 | 74.6 | 74.3 | 77.6 | 81.8 |
| WeRef rejection rate | 18.3 | 70.7 | 43.3 | 73.0 | 96.4 |
| PixMo-Count acc | n/a | 80.4 | 71.9 | 82.8 | 89.6 |
| TotalText OCR (F1) | 35.4 | n/a | 40.6 | 43.3 | 52.0 |
| SROIE OCR (F1) | 43.3 | n/a | 44.8 | 40.3 | 68.1 |
On matting (MAD ×10³, lower is better), WeVisOne-VL-4B scores 9.8 on Tencent's WeRefMat referring benchmark against 22.4 for a Qwen3.5-9B + SAM2-Matting pipeline. On ReasonSeg val it gets 71.2 gIoU vs 65.0 for a fine-tuned LISA-13B.
The rejection numbers are the ones I care about. Qwen3-VL-8B on the WeRef set rejects absent targets 18.3% of the time, which means it draws a box around something that isn't there most of the time. At 96.4% you can put the detector in a pipeline; at 18.3% you end up babysitting it. One caveat: WeRef-Bench and WeRefMat are Tencent's own benchmarks.
The 2B sibling is closer than you'd expect: 53.3 on LVIS, 92.9% WeRef rejection, 88.1 on PixMo-Count. If the 4B doesn't fit your card at bf16, the 2B through the official package is the one to try before you give up on masks.
I do have complaints about the comparison. Qwen3-VL-4B isn't in the tables. This model is a 4B built on Qwen3-VL, and Tencent compares it with the 8B. I'd like to know how much of the gain comes from the fine-tuning and how much from the base, and the one baseline that would tell us is missing.
There's no Gemma and no general VLM benchmark either: no DocVQA, no OCRBench, no MMMU, no chart QA. That fits what the model is for, but it leaves the "vs Qwen VL and Gemma" question unanswered for general image chat, and nobody has published those numbers yet.
People are already questioning the baseline selection. Discussion #3 on the model page asks how the baselines were selected, pointing out that the GroundingPI paper reports additional baselines and higher scores on some overlapping benchmarks, and how the model compares with frontier proprietary models. No reply at the time of writing.
How much VRAM it needs: bf16 vs Q8_0 vs Q4_K_M
Tencent publishes no VRAM figures. The table below is estimated from file sizes. These are weight sizes only; activations and KV cache come on top.
| Format | Weights | Vision projector | Total on disk | 8 GB card? |
|---|---|---|---|---|
| f16 GGUF | 8.84 GB | 0.84 GB (f16 mmproj) | ~9.7 GB | No |
| Q8_0 GGUF | 4.70 GB | 0.45 GB (Q8_0 mmproj) | ~5.2 GB | Yes |
| Q4_K_M GGUF | 2.72 GB | 0.45 GB | ~3.2 GB | Easily |
Sizes are the actual file sizes in mradermacher/WeVisOne-VL-4B-GGUF; the repo's README table rounds them up (2.8 GB for Q4_K_M, 0.6 GB for the Q8_0 mmproj). The Hub lists the original checkpoint at 5B params in BF16, so the official bf16 path is roughly 10 GB of weights. That doesn't fit on an 8 GB card without offloading.
There is one measured data point, from the PALLER-NOVUS GGUF repo: in a 20-query pilot on an RTX 2060 6 GB (context 4096, 1,024 to 1,960 image tokens), 4B Q4_K_M peaked at 4.38 GiB of whole-device VRAM at 4.17 s per request. The 4B F16 GGUF had to spill onto the CPU on that card and took about 60 s per query. My guess is that 12 GB is the realistic floor for the Tencent package at bf16, but nobody has published a measured number yet.
Run it with the official package (the only way to get masks)
The Tencent package is what loads mask_decoder.pt. Install it from the repo:
git clone https://github.com/Tencent/WeVisOne-VL.git && cd WeVisOne-VL
conda create -n wevisone-vl python=3.10 -y && conda activate wevisone-vl
pip install -r requirements.txt && pip install -v -e .
from PIL import Image
from wevisone_vl import WeVisOneVL
from wevisone_vl.visualization import save_result
model = WeVisOneVL("tencent/WeVisOne-VL-4B", backend="transformers") # or backend="vllm"
image = Image.open("demo/examples/images/seabreeze.jpg").convert("RGB")
result = model.predict(image, task="matting", query="person")
save_result("outputs/matting", image, result)
That's the README example. Swap task for detection (with a list of categories as query), ocr, count_box and so on. The README also lists options for device ("auto", "cuda:0", "npu:0", "cpu"), dtype, min_pixels / max_pixels and the usual decoding arguments like max_new_tokens and temperature. If you're short on memory, lower max_pixels before anything else; fewer visual tokens means less activation memory. There's a Gradio demo too:
CUDA_VISIBLE_DEVICES=0 python demo/app.py --model tencent/WeVisOne-VL-4B
Add --backend vllm to run it on vLLM instead of transformers.
The Hub's "Use this model" menu also offers a plain vllm serve "tencent/WeVisOne-VL-4B" and an SGLang launch command. Since the architecture is stock qwen3_vl, those serve the language model only: you get coordinate tokens back, parse them yourself, and never see a mask.
The Hub's transformers snippet for the model asks "What animal is on the candy?" That's the generic image-text-to-text example, so don't judge the model by free-form chat; it was trained for structured outputs.
GGUF in Ollama and llama.cpp: boxes yes, masks no
There are three community GGUF repos for the 4B so far: mradermacher, prithivMLmods and PALLER-NOVUS, which ships 2B and 4B in one repo. The Hub's "Use this model" menu on the mradermacher repo gives:
llama-server -hf mradermacher/WeVisOne-VL-4B-GGUF:Q4_K_M
ollama run hf.co/mradermacher/WeVisOne-VL-4B-GGUF:Q4_K_M
The problem is that a GGUF is the language model plus an mmproj vision projector. The pixel decoder is a separate PyTorch file, and neither llama.cpp nor Ollama knows what to do with an <OBJ> token. So on an 8 GB card you should get detection, referring boxes, pointing, counting and OCR grounding as coordinate text, but not segmentation or matting. PALLER-NOVUS says the same: putting mask_decoder.pt next to a GGUF doesn't give you pixel output. Their Q4_K_M runs did return structured boxes on a pinned, patched llama.cpp build (they say the patch carries changes for another model and adds no WeVisOne capabilities, but stock upstream is untested), launched with --special so the coordinate tokens survive decoding. I haven't seen anyone confirm that Ollama's hf.co pull picks up the separate mmproj for this architecture, so test image input there before you build on it. There are no imatrix quants yet either; mradermacher only has static ones so far.
Is it the vision model to run on an 8 GB card?
For grounding work, I think yes. If your local pipeline is GroundingDINO for boxes plus SAM for masks, or a general VLM you keep asking for coordinates, this is the first small model I'd test as a replacement, mostly because of the rejection rate. Q4_K_M plus its projector at ~3.2 GB leaves plenty of room.
For "read this screenshot and tell me what the error says", I wouldn't switch. The OCR benchmarks here measure finding and transcribing text regions with boxes, not answering questions about a document. Stay on whatever general Qwen3-VL build you already use until someone posts DocVQA or OCRBench numbers for this one.
And if you want masks on 8 GB, you're waiting on a quantized path for the official package, or on someone wiring the pixel decoder into llama.cpp.
Which small vision model do you actually use for OCR or screenshots locally right now, and would you trust a 4B model's boxes without a human checking them?
Sources
- tencent/WeVisOne-VL-4B model card and config.json
- Tencent/WeVisOne-VL on GitHub (benchmark tables, install, tasks)
- WeVisOne-VL paper, arXiv:2610.31329
- mradermacher/WeVisOne-VL-4B-GGUF
- prithivMLmods/tencent-WeVisOne-VL-4B-GGUF
- PALLER-NOVUS/WeVisOne-VL-GGUF (RTX 2060 pilot in BENCHMARKS.md)
- Discussion #3: Questions on Baseline Selection
Mentions
Keep mask_decoder.pt alongside the model weights when loading WeVisOne-VL from a local folder The official wrapper downloads it for you when you pass a Hugging Face model ID github.com/Tencent/WeVisOne-VL/blob/main/wevisone…
we gave our AI an eye for detail. 👀 Meet WeVisOne-VL, an open-source 2B/4B model from our Weixin Vision team. It finds and counts objects, reads text, and makes precise masks and cutouts—no separate SAM needed. Not in the image? "None." Tencent/WeVisOne-VL
Serving WeVisOne-VL-4B: two engines, two copies of the weights, one chess piece apart Before you put WeVisOne-VL-4B behind an API, decide which engine writes the answer. The authors' package runs the model two ways: through vLLM, a server built for throughput, or through plain Hugging Face Transformers as the reference. That choice sets your memory, your latency and, on one test image, whether the count was right. The model comes from the WeChat Vision team at Tencent, on top of Alibaba's Qwen3-VL-4B-Instruct. One 4B checkpoint does the work of a detector, a segmenter and a matting tool. It boxes what you name, counts, points, reads text with its position, outlines and cuts out. We deployed it on our own cloud and sent the same 14 requests to each setup. Tables are in the full WeVisOne-VL-4B test with every measurement. Why the fast path holds the model twice Language models write tokens, word pieces produced one at a time. This one also has position tokens: four make a box, two a point. For outlines and cut-outs it writes a marker token after each box. A separate cutter of 9.6 million parameters (mask_decoder.pt) then reads the model's internal state at that marker and paints the pixels. vLLM hands back the text, not those internal states. So with backend="vllm" the package also loads the full Transformers model and replays the answer through it for the cutter. At half precision the weights weigh about 10 GB, and this path keeps two copies. On an RTX 4090 it peaked at 23.0 GiB of 24 GB, against 12.3 GiB for Transformers alone. The default settings do not start on 24 GB With the package's defaults on a 4090, vLLM refuses to boot. It reserves room for ten images and ten videos per prompt, and that leaves nothing for the KV cache, the scratch memory where a model keeps what it has already read and written: Encoder cache will be initialized with a budget of 16384 tokens Available KV cache memory: -0.87 GiB ValueError: No available memory for the cache blocks Our custom backend, the small Python program our platform runs on the GPU machine instead of a standard model server, wraps vllm.LLM with one image per prompt, no video, a shorter context (the most tokens one request may hold) and at most four requests in flight. Giving vLLM 0.5 of the card next to the second copy made the big portrait cut-out run out of memory. At 0.465 with a 4096-token context, the cache came up short. At 0.465 with 3072 tokens, all 14 requests passed. Thin margin: measure peak memory with your own image sizes. Same greedy decoding, different answers On the authors' chess photo (16 white pieces), vLLM said 15 in 1.35 seconds, missing the knight on g5. Transformers said 16 in 3.87 seconds. !Side by side, WeVisOne-VL-4B counting white chess pieces with two engines on an RTX 4090. Left, vLLM: 15 in 1.35 seconds, the white knight on g5 marked as missed. Right, Transformers: 16 in 3.87 seconds, the knight boxed Both runs used greedy decoding, meaning the model always takes its single most likely next token, so no dice are rolled. Most likely the two programs round their maths a little differently, and a box on the edge tipped over. They also split on a street sign (29 words against 21) and on hanging pots (19 against 20). One wrong count out of 14 says nothing about a rate. It does say: run your evaluation set on the engine you will ship. What to measure before you ship - The first request after a start. On vLLM, a detection that runs in 1.9 seconds once warm took 14.5 to 15.5 seconds the first time. Fire a throwaway request at boot. - Peak memory per task, not per model. The cutter asks for about 0.8 GB at once, at any photo size. With the 4090 capped like a strict 12 GB card, the text jobs (boxes, counts, points, reading) still passed. Outlines and cut-outs all failed, even on a 640-pixel photo. - Response size. The platform dropped results above 8 MB, and the caller waited for a 504, the HTTP gateway-timeout error. The backend now fits a 5.2 MB budget (JPEG overlay, 1280 px cut-out). The platform side, where the client waits out the full timeout, is open. The reference engine is the safer default The smallest card that passed all 14 requests was a real RTX 5080 with 16 GB, on Transformers only, peaking at 12.8 GiB. Against the 4090's fast engine it took roughly double the time on the big cut-out (2.1 seconds for a 1620 by 1080 portrait, against 1.0) and up to four times as long on object jobs. The backend picks vLLM on cards with 22 GiB or more in total, Transformers below. Release 0.1.3 makes the tested values the defaults, but has not itself been run on a GPU. 0.1.0 to 0.1.2 were, with the values passed by hand. The 12 GB and 8 GB figures are caps applied on the 4090, not real smaller cards. We sent one request at a time, so concurrency is untested. The authors claim the lead on over 30 benchmarks (55.3 against GroundingDINO's 38.8 on finding named objects). We did not rerun them; we judged outputs by eye and counted the chessboard by hand. The WeVisOne-VL-4B deployment write-up on inferencekey.com has the timing table, the VRAM ladder down to an 8 GB cap with the 2B model, the environment variables and each problem's status. The code is in the SDK example for WeVisOne-VL-4B. Credit: Tencent's WeChat Vision team built the model, the code package and the model card. Qwen3-VL underneath is Alibaba's. The vLLM and Transformers projects make both engines, and the AI Search channel's news video put the model on our radar. Our part was deploying it, measuring it and writing the backend. Read the complete WeVisOne-VL-4B results, tables and setup before you size your card.
How to Run Tencent WeVisOne-VL-4B Locally on 8 GB VRAM (Ollama Drops the Masks) Tencent's WeVisOne-VL-4B picked up 5,986 Hugging Face downloads last month, and I suspect a lot of those people expected another small chat-about-your-image model. It's a perception model instead: boxes, points, counts, text locations, segmentation masks and alpha mattes, all from one 4B checkpoint built on Qwen3-VL. You can run WeVisOne-VL-4B locally on an 8 GB GPU with a Q4_K_M GGUF in llama.cpp, and the Hub offers a one-line Ollama pull too. You just lose the masks when you do. I went through the model card, the GitHub README, the config and the GGUF repos. This post covers what the model is, Tencent's numbers against Qwen3-VL-8B (and the baseline they left out), VRAM per format, and the two ways to run it. What WeVisOne-VL-4B actually is The model card says it comes from WeChat Vision at Tencent and is built on Qwen3-VL. The config.json confirms it: Qwen3VLForConditionalGeneration, 36 text layers, hidden size 2560, a 24-layer ViT with patch size 16. That's the Qwen3-VL-4B layout with a 152,680-token vocabulary. The extra tokens are the trick. Coordinates are emitted as <0> to <1000> (normalized boxes and points), and every predicted box in dense tasks is followed by an <OBJ> token (ID 152678). A separate 9.6M-parameter pixel decoder, shipped as mask_decoder.pt, reads those <OBJ> positions plus multi-level ViT features and produces masks and stride-1 alpha mattes, so there's no SAM in the loop. The task list from the GitHub repo: detection, referring, point, count_box / count_point, ocr / ocr_grounding, segmentation, matting. Other facts worth knowing before you download: - License: Apache-2.0 for Tencent-authored code; third-party components keep their own licenses. - Context: max_position_embeddings is 262144, same as the Qwen3-VL base. Tencent's evals cap images at 1,960 visual tokens, so context isn't the constraint here. - Single-turn, single image. The pixel decoder rejects video grids, and the model returns structured output without chain-of-thought. - It answers None when the thing you asked for isn't in the image, which matters more than it sounds (see the benchmarks). - There's also a 2B sibling. Paper is arXiv:2610.31329, submitted 22 Oct 2026. WeVisOne-VL-4B vs Qwen3-VL benchmarks, and who's missing All numbers below are from the GitHub README. Tencent compares against Qwen3-VL-8B, Qwen3.5-9B, Rex-Omni-3B, LocateAnything-3B and a few specialists. | Task (metric) | Qwen3-VL-8B | Qwen3.5-9B | Rex-Omni-3B | LocateAnything-3B | WeVisOne-VL-4B | |---|---|---|---|---|---| | LVIS detection (F1) | 42.9 | 43.7 | 46.9 | 50.7 | 55.3 | | Dense200 detection (F1) | 13.8 | 18.0 | 58.3 | 58.7 | 71.3 | | RefCOCOg test (F1) | 74.9 | 74.6 | 74.3 | 77.6 | 81.8 | | WeRef rejection rate | 18.3 | 70.7 | 43.3 | 73.0 | 96.4 | | PixMo-Count acc | n/a | 80.4 | 71.9 | 82.8 | 89.6 | | TotalText OCR (F1) | 35.4 | n/a | 40.6 | 43.3 | 52.0 | | SROIE OCR (F1) | 43.3 | n/a | 44.8 | 40.3 | 68.1 | On matting (MAD ×10³, lower is better), WeVisOne-VL-4B scores 9.8 on Tencent's WeRefMat referring benchmark against 22.4 for a Qwen3.5-9B + SAM2-Matting pipeline. On ReasonSeg val it gets 71.2 gIoU vs 65.0 for a fine-tuned LISA-13B. The rejection numbers are the ones I care about. Qwen3-VL-8B on the WeRef set rejects absent targets 18.3% of the time, which means it draws a box around something that isn't there most of the time. At 96.4% you can put the detector in a pipeline; at 18.3% you end up babysitting it. One caveat: WeRef-Bench and WeRefMat are Tencent's own benchmarks. The 2B sibling is closer than you'd expect: 53.3 on LVIS, 92.9% WeRef rejection, 88.1 on PixMo-Count. If the 4B doesn't fit your card at bf16, the 2B through the official package is the one to try before you give up on masks. I do have complaints about the comparison. Qwen3-VL-4B isn't in the tables. This model is a 4B built on Qwen3-VL, and Tencent compares it with the 8B. I'd like to know how much of the gain comes from the fine-tuning and how much from the base, and the one baseline that would tell us is missing. There's no Gemma and no general VLM benchmark either: no DocVQA, no OCRBench, no MMMU, no chart QA. That fits what the model is for, but it leaves the "vs Qwen VL and Gemma" question unanswered for general image chat, and nobody has published those numbers yet. People are already questioning the baseline selection. Discussion #3 on the model page asks how the baselines were selected, pointing out that the GroundingPI paper reports additional baselines and higher scores on some overlapping benchmarks, and how the model compares with frontier proprietary models. No reply at the time of writing. How much VRAM it needs: bf16 vs Q8_0 vs Q4_K_M Tencent publishes no VRAM figures. The table below is estimated from file sizes. These are weight sizes only; activations and KV cache come on top. | Format | Weights | Vision projector | Total on disk | 8 GB card? | |---|---|---|---|---| | f16 GGUF | 8.84 GB | 0.84 GB (f16 mmproj) | ~9.7 GB | No | | Q8_0 GGUF | 4.70 GB | 0.45 GB (Q8_0 mmproj) | ~5.2 GB | Yes | | Q4_K_M GGUF | 2.72 GB | 0.45 GB | ~3.2 GB | Easily | Sizes are the actual file sizes in mradermacher/WeVisOne-VL-4B-GGUF; the repo's README table rounds them up (2.8 GB for Q4_K_M, 0.6 GB for the Q8_0 mmproj). The Hub lists the original checkpoint at 5B params in BF16, so the official bf16 path is roughly 10 GB of weights. That doesn't fit on an 8 GB card without offloading. There is one measured data point, from the PALLER-NOVUS GGUF repo: in a 20-query pilot on an RTX 2060 6 GB (context 4096, 1,024 to 1,960 image tokens), 4B Q4_K_M peaked at 4.38 GiB of whole-device VRAM at 4.17 s per request. The 4B F16 GGUF had to spill onto the CPU on that card and took about 60 s per query. My guess is that 12 GB is the realistic floor for the Tencent package at bf16, but nobody has published a measured number yet. Run it with the official package (the only way to get masks) The Tencent package is what loads mask_decoder.pt. Install it from the repo: git clone https://github.com/Tencent/WeVisOne-VL.git && cd WeVisOne-VL conda create -n wevisone-vl python=3.10 -y && conda activate wevisone-vl pip install -r requirements.txt && pip install -v -e . from PIL import Image from wevisone_vl import WeVisOneVL from wevisone_vl.visualization import save_result model = WeVisOneVL("tencent/WeVisOne-VL-4B", backend="transformers") # or backend="vllm" image = Image.open("demo/examples/images/seabreeze.jpg").convert("RGB") result = model.predict(image, task="matting", query="person") save_result("outputs/matting", image, result) That's the README example. Swap task for detection (with a list of categories as query), ocr, count_box and so on. The README also lists options for device ("auto", "cuda:0", "npu:0", "cpu"), dtype, min_pixels / max_pixels and the usual decoding arguments like max_new_tokens and temperature. If you're short on memory, lower max_pixels before anything else; fewer visual tokens means less activation memory. There's a Gradio demo too: CUDA_VISIBLE_DEVICES=0 python demo/app.py --model tencent/WeVisOne-VL-4B Add --backend vllm to run it on vLLM instead of transformers. The Hub's "Use this model" menu also offers a plain vllm serve "tencent/WeVisOne-VL-4B" and an SGLang launch command. Since the architecture is stock qwen3_vl, those serve the language model only: you get coordinate tokens back, parse them yourself, and never see a mask. The Hub's transformers snippet for the model asks "What animal is on the candy?" That's the generic image-text-to-text example, so don't judge the model by free-form chat; it was trained for structured outputs. GGUF in Ollama and llama.cpp: boxes yes, masks no There are three community GGUF repos for the 4B so far: mradermacher, prithivMLmods and PALLER-NOVUS, which ships 2B and 4B in one repo. The Hub's "Use this model" menu on the mradermacher repo gives: llama-server -hf mradermacher/WeVisOne-VL-4B-GGUF:Q4_K_M ollama run hf.co/mradermacher/WeVisOne-VL-4B-GGUF:Q4_K_M The problem is that a GGUF is the language model plus an mmproj vision projector. The pixel decoder is a separate PyTorch file, and neither llama.cpp nor Ollama knows what to do with an <OBJ> token. So on an 8 GB card you should get detection, referring boxes, pointing, counting and OCR grounding as coordinate text, but not segmentation or matting. PALLER-NOVUS says the same: putting mask_decoder.pt next to a GGUF doesn't give you pixel output. Their Q4_K_M runs did return structured boxes on a pinned, patched llama.cpp build (they say the patch carries changes for another model and adds no WeVisOne capabilities, but stock upstream is untested), launched with --special so the coordinate tokens survive decoding. I haven't seen anyone confirm that Ollama's hf.co pull picks up the separate mmproj for this architecture, so test image input there before you build on it. There are no imatrix quants yet either; mradermacher only has static ones so far. Is it the vision model to run on an 8 GB card? For grounding work, I think yes. If your local pipeline is GroundingDINO for boxes plus SAM for masks, or a general VLM you keep asking for coordinates, this is the first small model I'd test as a replacement, mostly because of the rejection rate. Q4_K_M plus its projector at ~3.2 GB leaves plenty of room. For "read this screenshot and tell me what the error says", I wouldn't switch. The OCR benchmarks here measure finding and transcribing text regions with boxes, not answering questions about a document. Stay on whatever general Qwen3-VL build you already use until someone posts DocVQA or OCRBench numbers for this one. And if you want masks on 8 GB, you're waiting on a quantized path for the official package, or on someone wiring the pixel decoder into llama.cpp. Which small vision model do you actually use for OCR or screenshots locally right now, and would you trust a 4B model's boxes without a human checking them? Sources - tencent/WeVisOne-VL-4B model card and config.json - Tencent/WeVisOne-VL on GitHub (benchmark tables, install, tasks) - WeVisOne-VL paper, arXiv:2610.31329 - mradermacher/WeVisOne-VL-4B-GGUF - prithivMLmods/tencent-WeVisOne-VL-4B-GGUF - PALLER-NOVUS/WeVisOne-VL-GGUF (RTX 2060 pilot in BENCHMARKS.md) - Discussion #3: Questions on Baseline Selection
Tencent open-sourced WeVisOne-VL: a single 2B/4B VLM for detection, referring, counting, OCR, segmentation and matting. Apache 2.0. They report 55.3 F1 on LVIS for the 4B, vs 50.7 for the best open baseline. Tencent/WeVisOne-VL
开源地址与在线体验： • GitHub 开源仓库：Tencent/WeVisOne-VL • 论文页面：huggingface.co • 模型权重合集：huggingface.co 当 4B 级别的原生 VLM 已经能直接输出高质量 Alpha 抠图和精准坐标，传统的独立检测与分割小模型还会保留多少生存空间？
Tencent just open-sourced WeVisOne-VL (2B/4B, Apache 2.0): one small vision model for detection, counting, OCR, segmentation and alpha mattes. Builders: swap 3 specialist vision tools for one local model and compare cost. Tencent/WeVisOne-VL