Shared Representations in Multi-Task Models: The Split Point as a Modelling Choice

Where two different streams of information are allowed to meet each other in a multi task or multi modal model seems to be a structural decision (ie, an architectural choice you make when you sketch the model out), but it is actually a modelling choice, with impact on what the model learns and how it trains. This post is an attempt to think through the various things that might influence this decision.

A neural network takes an input and passes it through a sequence of layers. Each layer produces a representation of that input — a vector of numbers that encodes something about it. A useful way to think about this is that early layers tend to capture low-level structure (in an image for example: edges, colour gradients, local texture), while later layers capture progressively higher-level abstractions (tissue architecture, gland morphology, cellular arrangement patterns – though this is somewhat of a toy example). By the time you reach the final layer, the representation has been compressed and transformed into something the network can use to make a prediction.

Now suppose you have two (or more!) inputs rather than one. In computational pathology, this comes up frequently; a common example is pairing an H&E whole slide image with a bulk RNA-seq profile from the same tissue sample. Both carry information about the tumour, but they are measuring fundamentally different things at different scales and resolutions. You want to build a single model that uses both.

The immediate question is: when do these two streams of information meet?

You can imagine two parallel processing towers — one for the image, one for the RNA-seq data — with a connection between them. The position of that connection, measured in terms of how deep into each tower’s processing it occurs, is the fusion point. Fuse early, and the two raw (or nearly raw) signals are merged almost immediately, leaving a single joint network to do the rest of the work. Fuse late, and each modality is processed almost entirely independently, with only the final representations combined before prediction.

A closely related concept appears in multi-task models, where a single input feeds a shared trunk that eventually diverges into separate heads, each producing a different output (say, tumour grade and a survival risk score). Here the relevant decision is the split point — how deep into the network the shared representation is maintained before the two tasks go their separate ways. The geometry is a mirror image of the fusion case, but the underlying question is the same: at what depth of representation does the connection happen?

The shared trunk as a constrained representation

In a single-task model, every layer is free to transform the input in whatever way best serves that one output. There are no competing demands. In a shared trunk — or equivalently, in the layers after a fusion point — something more complex is happening because those shared layers have to produce a single representation that is simultaneously useful for two downstream objectives. That is a constraint because every weight in the shared portion of the network is updated based on signals from both tasks or both modalities, and it cannot serve one without also serving the other.

A useful way to think about this is through the lens of mutual information. The shared representation $Z$ , derived from inputs $X_1$ and $X_2$ , needs to retain enough information about both target variables $Y_1$ and $Y_2$ for the downstream heads to work from:

I(Z; Y_1) > 0 \; \text{and} \; I(Z; Y_2) > 0

where $I(\cdot)$ denotes mutual information. If $Y_1$ and $Y_2$ depend on genuinely different features of the inputs, the shared representation is being asked to preserve two distinct signals simultaneously. Each layer acts as a kind of compression — only information that is useful for the eventual output tends to survive. A shared trunk tightens that compression, because it has to serve two outputs rather than one. Something can get deprioritised, and which signal that is depends on the relative strength of the gradients arriving from each head during training.

This is not necessarily a flaw in the approach — shared representations are often the point, because the model can discover structure that is useful for both tasks at once. But it does mean that the amount of sharing, and where it happens, is an important decision.

Moving the dial

Let’s think of the fusion or split point as a dial you can turn from one end to the other. What changes as you move it?

Late fusion (each modality processed independently, combined only at the end)

With late fusion, each processing tower runs almost entirely independently. The image encoder learns image representations; the RNA-seq encoder learns molecular representations. Each is free to optimise for its own input without interference from the other. Only the final, high-level representations are combined before the prediction is made.

The advantage is that each encoder can develop rich, task-relevant representations of its own modality. The limitation is that the model can only combine high-level abstractions — it cannot discover interactions that are only apparent when looking at intermediate features of both modalities together. If the relationship between morphology and molecular signal is complex and conditional on lower-level features, late fusion will miss it.

Early fusion (modalities combined near the input, processed jointly thereafter)

With early fusion, the raw or near-raw signals from both modalities are concatenated and passed through a single joint network. The model has maximum capacity to learn cross-modal interactions at every level of representation.

The risk is that with no period of independent processing, the model is asked to simultaneously learn what is meaningful in each modality and how the two modalities interact. On smaller datasets — which are common in computational pathology — this is a lot to ask. The model may latch onto co-occurrences in the training data that do not generalise, rather than learning meaningful structure in either modality independently.

Intermediate fusion

Somewhere between these two extremes, each modality is processed up to some representational depth, and then the streams are merged. The depth at which the merge happens determines the level of abstraction at which cross-modal interaction is permitted.

This is where the modelling assumption becomes most legible. By choosing an intermediate fusion point, you are effectively saying: “The representations of these two modalities become meaningfully relatable at this level of abstraction.” That is a prior about the structure of the problem — and like all priors, it can be right or wrong.

What the gradients are doing

We’ve been talking about how fusion/split points impact model training by affecting the learned representation according to the output task. Concretely, we know that this representation is optimised through backpropagation, so let’s take a look at what the gradients are doing.

During backpropagation, the loss from each head or output produces a gradient that flows backwards through the network. Above the split point — in the head-specific layers — each task’s gradients flow independently, updating only the weights relevant to that task. Below the split point, in the shared trunk, gradients from both tasks arrive and are summed:

\frac{\partial L}{\partial \theta_{shared}} = \frac{\partial L_1}{\partial \theta_{shared}} + \frac{\partial L_2}{\partial \theta_{shared}}

When the two gradient contributions point in similar directions, the shared weights update cleanly — both tasks are pulling the representation toward the same region of parameter space. When they conflict, the shared weights receive a contradictory signal. The update that improves the representation for task one may degrade it for task two, and the shared layers can end up oscillating or settling at a compromise that serves neither task particularly well.

The depth of the shared trunk determines how many layers are exposed to this conflict. A shallow shared trunk — corresponding to an early split or late fusion — limits the number of layers receiving conflicting gradients. A deep shared trunk propagates the conflict through most of the network. This is part of the practical motivation behind methods like uncertainty-weighted loss (Kendall et al., 2018), which learns to balance gradient contributions from each task during training, and gradient surgery approaches like PCGrad, which explicitly project conflicting gradients to reduce interference. Knowing where your split point sits gives you a better sense of whether these methods are likely to be relevant for your problem.

Bringing this back to Computational Pathology

H&E morphology and bulk RNA-seq are a natural pair to think through here, partly because they are commonly used together and partly because they have a structural asymmetry that makes the fusion point question concrete.

Morphology is spatially resolved and local. An image patch captures cellular detail, nuclear morphology, and tissue microarchitecture at a specific location on the slide. Bulk RNA-seq, by contrast, is spatially averaged — it measures gene expression across the entire tissue sample, collapsing spatial variation into a single profile. The two signals are not operating at the same scale.

This asymmetry has a direct implication for where fusion makes sense. If you fuse early — connecting pixel-level or patch-level image features to a tissue-averaged molecular profile — you are asking the model to relate things that are not yet at comparable levels of abstraction. The image features are local; the RNA-seq features are global. The model has to implicitly perform the aggregation from local to global as part of learning the cross-modal relationship, which is a significant additional burden.

If instead the image encoder is allowed to aggregate local patch features into something more tissue-level — through attention-based pooling or multiple instance learning, for instance — before the fusion point, the two representations are in more comparable abstract spaces when they meet. Both are now describing the tissue at roughly the same scale. The fusion is easier to learn because the two streams are speaking a more similar language at the point they connect.

This adds a second dimension to the fusion point decision. It is not only about task relatedness — about whether the two outputs benefit from shared representations. It is also about representational commensurability: whether the two streams have been processed to the point where they are describing the input at a compatible level of abstraction. A fusion point that satisfies both of these considerations simultaneously is often not at the extremes of the dial.

Closing thought

The fusion or split point tends to get decided early and revisited rarely. It looks like scaffolding. Thinking of it instead as an explicit prior — about when modalities interact, and at what scale of abstraction they become comparable — makes it easier to reason about whether the choice fits the problem, and what to look for if the model is not behaving as expected.

Enjoyed this post?

If you found this useful and want to read more on AI, machine learning, and computational pathology, subscribe to the newsletter below. New posts go out (kind of) weekly and cover everything from the mathematics of image analysis to practical ML workflows for digital pathology.