If you’re going to be doing medical image processing research as someone without any formal medical training, then you ought to keep medical professionals involved in your work. This should seem obvious – how on earth is one supposed to validate the performance of their deep learning models on their medical imaging task, if they can’t understand the ground truths themselves? One might leave it up to pure metrics – yes the segmentation accuracy is high because the IoU is good, or your confusion matrix metrics seem alright, but this isn’t the point! Real progress in methods and approaches are made when you can identify when and where your model is going wrong, not just when it’s going right. In a medical imaging context, this can really only be done by trained professionals, who have the domain expertise to truly dig into model predictions.
I look at deep learning for histological diagnosis of cancer, and I’m very lucky to have two histopathologists on my PhD supervisory team. They add context and keep the work I do grounded in their own clinical contexts, ultimately making what I do more useful for the domain I’m working in. The problem in histopathology however, is that it’s not exactly trivial to be able to share model predictions. Whole-slide images (the digitised microscope scans we work with) are multi gigapixel in size, making full-resolution files impractical to send or open in any conventional way. You can’t just compress them either, because histopathologists need to zoom into cellular details to actually understand what the model is doing. And static images simply don’t cut it: without the ability to pan, zoom, toggle overlays, or adjust thresholds, there’s no real way to explore where predictions are failing. Moreover, on this point, the software that histopathologists use in the lab have these dynamic features, so the more comfortable they feel with what I share, the easier it is for them to give me feedback. This is important as they are really busy people!
This is why I built a small web app to make whole-slide image model outputs explorable in the browser. It uses deep-zoom tiling (which is the same approach Google Maps uses, more on this later) so viewers only load the visible portion of a slide at the resolution they need. Prediction heatmaps can be toggled on and off, opacity adjusted, and ground truth overlaid. A threshold slider highlights false positives and false negatives, letting you see exactly where the model over- or under-calls. It runs on any modern browser with no special software or giant downloads, making it easy to share with clinical collaborators and supervisors
Why did I build this instead of using already available viewers (like ASAP, Pathcore, etc)? A few reasons:
- I have a Macbook with a newer chip, so software like ASAP is not downloadable on my device for now;
- Commercial solutions are extremely expensive, and outside the budget for my PhD funding and;
- I wanted the custom features of overlaying the prediction heatmaps and toggling predictions over my test slides, and thought the easiest way to do this was to just build it from scratch.
If you’re interested in the tech stack and design choices please read on. Otherwise, please find the viewer here, loaded with a few test slides from CAMELYON16 and predictions were made by the model I discussed here.
The Build
The code for this viewer can be found here . Here is some more detail about the build:
The fundamental challenge with whole-slide images is their size — we’re talking gigapixels. You can’t just throw that into an <img> tag and call it a day. The solution is the same one that Google Maps uses: tile pyramids.
Instead of loading one enormous image, we pre-slice it into a pyramid of tiles at multiple zoom levels. At the lowest zoom, maybe 4 tiles cover the whole slide. At the highest, thousands. The viewer only ever fetches what’s currently visible on screen, at the appropriate resolution. Pan left? Fetch those tiles. Zoom in? Fetch higher-resolution tiles for that region. The format that enables this is Deep Zoom Image (DZI), which is an XML manifest pointing to a folder hierarchy of PNGs.
The Processing Pipeline
Everything starts with a shell script that loops over my test slides:
WSI (.tif)
↓ vips dzsave
Base tiles (DZI pyramid)
↓
Predictions pickle + base.dzi
↓ make-overlay.py
Heatmap PNG + gt_patches.json + patches_index.json
↓ vips dzsave
Overlay tiles (DZI pyramid)
↓
manifest.json (ties it all together)vips handles the tiling — it’s a fast, memory-efficient image processing library that can handle massive files without loading them entirely into RAM. Each slide ends up as two tile pyramids (base + prediction overlay) plus some lightweight JSON for ground truth coordinates and per-patch probabilities.
The Python script does the interesting bit: it reads my model’s output (a pickle with patch coordinates and probabilities), rasterizes them onto a canvas matching the slide dimensions, applies a jet-style colormap, and writes out both the heatmap PNG and the JSON indices needed for interactive error analysis.
The Frontend
The viewer is vanilla JavaScript — no React, no build step, just a single app.js file. The heavy lifting comes from OpenSeadragon, a mature library for deep zoom viewing. I feed it the base DZI, then add the prediction heatmap as a second tiled layer with adjustable opacity.
The ground truth and FP/FN overlays work differently. Rather than pre-rendering them as tiles (which would balloon storage for every possible threshold), they’re drawn client-side onto HTML canvas elements that sit above the OpenSeadragon viewer. The browser reads the JSON indices, and on every pan/zoom event, redraws only the patches currently in view. This keeps the tile storage fixed while letting users adjust the classification threshold on the fly. The heatmap colormap is replicated in JavaScript to draw a matching legend.
Hosting
There’s no backend. The entire app is:
- Tiles + JSON → S3 bucket, served via CloudFront CDN
- HTML/JS/CSS → GitHub Pages
A pathologist in London and one in Sydney both get tiles served from edge nodes near them. First load fetches maybe 50KB of JS/CSS, then tiles stream in as needed. I can share a link and it just works — no installs, no accounts, no waiting for a 2GB file to download.
The only “deployment” is syncing the tiles folder to S3 and pushing the frontend to GitHub. Cost is essentially zero for my usage — CloudFront charges for bandwidth, but tile requests are tiny and cached aggressively.
Design Decisions
A few choices that shaped the architecture:
Tile size of 256px — standard for deep zoom, balances request overhead against file size. Matches what OpenSeadragon expects by default as well as the tile size at which I trained my model
Predictions at level 2 — my model runs on patches from level 2 of the WSI pyramid (10x downsampled from full resolution).
JSON indices instead of baked-in thresholds — I wanted clinicians to experiment with different probability cutoffs without regenerating tiles. Storing [y, x, probability, is_tumour] for each patch means the threshold slider works instantly, purely in the browser.
Mask-based alpha rather than value-based — early versions made heatmap opacity proportional to probability, but pathologists found it hard to see low-confidence predictions. Constant alpha (around 70%) inside the prediction mask gives uniform visibility; the colour still encodes probability.
No framework — the UI is simple enough that vanilla JS + CSS keeps things maintainable and fast. The drawer panel, toast messages, and legend are all just a few dozen lines of CSS. Adding React would’ve meant a build step, node_modules, and complexity that wasn’t paying for itself.
Tradeoffs
The main downside of this approach is that everything has to be pre-computed. Every slide needs to go through the tiling pipeline before it can be viewed — for my test set of just a few slides, that’s a couple of hours of processing and a few gigabytes of tiles to upload to S3. There’s no “upload a slide and see predictions instantly” workflow here. If I retrain a model and want to visualise the new outputs, I’m re-running the pipeline and re-uploading. For my use case — sharing a fixed set of results with supervisors — this is fine. But it wouldn’t scale to a scenario where you need rapid iteration or on-demand inference. A proper production system would need a tile server that generates tiles lazily, or a backend that runs inference in real-time. That’s a different beast entirely, and for a PhD project, pre-baked tiles hit the sweet spot of simplicity and performance.
_______________________________________________________________________________________
If you found this interesting and would like to see more about how I’m applying computer vision and ML concepts to my PhD in Medical Image Processing, please do subscribe here:
Leave a Reply