A custom dataset of 2,520 images
Five instrument categories across three viewpoints, four backgrounds, and two hinge states, all captured under controlled variation and published openly on Hugging Face.
Bachelor Thesis / Leiden University
Active Vision for Surgical Instrument Recognition
I developed a computer vision system that recognizes surgical instruments and determines whether hinged tools are open or closed. By actively selecting the most informative camera views, the system improved recognition performance while reducing the number of images required.
Leiden University · BSc Data Science & AI
About this project
This was my BSc capstone at Leiden University's Institute of Advanced Computer Science, supervised by Dr. Daan Pelt and Dr. Jiayang Shi. The starting point was a simple observation: most surgical computer vision research focuses on minimally invasive surgery, where cameras are fixed and scenes are controlled. Open surgery is messier, and reliable tool recognition there remains unsolved.
What I Built
A two-stage recognition pipeline
A YOLO-based object detector identifies and localizes the instrument with 98.2% mAP. For hinged tools, an EfficientNet-B0 classifier predicts open vs. closed with 98.1% accuracy on cropped images.
An active view acquisition strategy
Instead of always using all three camera angles, the system starts with a single top-down view and only requests additional viewpoints when its prediction confidence is insufficient, saving acquisition effort while maintaining accuracy.
A comparative VLM evaluation
Six zero-shot vision-language models, including Gemma, Qwen3-VL, and Ministral variants, were benchmarked on the same recognition tasks to test whether general-purpose AI could substitute for a trained specialist model.
The Key Finding
The most important finding was that different tasks benefit from different viewpoints.
Category recognition
Works best from a close-up view. Distinguishing a needle holder from a clamp depends on fine local detail: jaw pattern, tip shape, and surface texture.
Hinge-state classification
Works best from a top-down view. Determining open vs. closed depends on global geometry: the angle between the two branches, not fine surface detail.
This means viewpoint selection cannot be task-agnostic: the best angle for identifying a tool is not necessarily the best angle for reading its state. It motivates a multi-view system and explains why an adaptive one can outperform simply fusing all views every time.
Dataset Samples
The dataset varies instrument type, viewpoint, background, and hinge state. These examples show the controlled visual variation used for training and evaluation.
Four Experiments
The project was evaluated through four experiments, moving from basic recognition to active multi-view decision making.
Can the system identify the instrument?
I first trained YOLO-based object detectors to recognize five surgical instrument categories: clamps, needle holders, scalpels, shears, and tweezers. The best detector reached 98.2% mAP@50:95, showing that trained vision-only models can recognize the instruments reliably in the controlled dataset.
- Key result
- 98.2% mAP@50:95
- Main takeaway
- Close-up views were most useful because they reveal fine details such as jaw shape, tip geometry, and surface texture.
Can it tell whether a hinged tool is open or closed?
For hinged instruments, I trained separate classifiers to predict whether the tool was open or closed. EfficientNet-B0 performed best, reaching 98.1% accuracy on cropped instrument images.
- Key result
- 98.1% accuracy
- Main takeaway
- Hinge-state recognition worked best from top-down views, where the global geometry and opening angle are easiest to see.
Should the system always use every camera view?
I then evaluated an active multi-view pipeline. The system starts from a top-down image and only requests another view when its confidence is too low, avoiding full three-view acquisition by default.
- Key result
- Highest accuracy using 2.5 views on average
- Main takeaway
- Active view acquisition sometimes improved over the initial single-view result while using fewer views than full three-view acquisition.
Can general-purpose VLMs replace trained specialist models?
Finally, I compared the trained vision-only pipeline with zero-shot vision-language models such as Qwen3-VL, Gemma, and Ministral. The best VLM reached 72.2% category accuracy and 62.4% hinge-state accuracy, but was much slower than the trained pipeline.
- Key result
- 72.2% category accuracy; 62.4% hinge-state accuracy
- Main takeaway
- VLMs were useful as a comparison baseline, but trained task-specific models remained better for fine-grained surgical instrument recognition.
Active View Acquisition
Confidence-driven acquisition avoids using every camera angle by default. The system asks for another view only when the current prediction is not confident enough.
Active Vision Results
The active system outperformed both single-view inference and full three-view fusion while using fewer acquisitions than the latter.
| Setting | Single view | All 3 views | Active |
|---|---|---|---|
| End-to-end accuracy | 55.0% | 70.0% | 73.3% |
| Mean views used | 1.0 | 3.0 | 2.5 |
Grad-CAM Analysis
These Grad-CAM heatmaps inspect the EfficientNet-B0 hinge-state classifier. Warmer regions mark the image areas that contributed most to an open or closed prediction. The strongest activations often appear around the hinge, handles, and branch angle, which matches the geometric nature of the task.
Open materials