Artificial Intelligence / Machine Learning

Layer-wise Relevance Propagation (LRP)

Meghana

Nov 28, 2023 • 6 min read

Introduction:

Layer-wise Relevance Propagation (LRP) is a model explainability technique used to interpret the predictions made by deep learning models. It provides insights into why a model makes certain decisions by attributing relevance scores to individual input features. LRP is particularly useful for understanding complex models such as neural networks, where the reasoning behind their predictions might not be immediately apparent.

Understanding LRP:

LRP works by redistributing the output relevance (prediction score) of a neural network back to its input features. This backward propagation of relevance helps us determine the contribution of each input feature towards the final prediction. The relevance scores are propagated through the layers of the neural network, and the sum of these relevance scores is equal to the model's output relevance.

Code Implementation of LRP in PyTorch:

Importing the Required Libraries

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

The necessary libraries are imported: torch, torch.nn, torch.nn.functional, and numpy.

Torch is the main PyTorch library, while torch.nn provides the tools for building neural network modules. torch.nn.functional contains various functions for building neural network layers, and numpy is used for numerical computations.

Creating a Sample Model

class SampleModel(nn.Module):
def init(self):
super(SampleModel,self).init()
self.fc1 = nn.Linear(10,32)
self.fc2 = nn.Linear(32,16)
self.fc3 = nn.Linear(16,1)

def forward(self,x):
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = torch.sigmoid(self.fc3(x))
return x

A sample neural network is a defined model called SampleModel by inheriting from nn.Module, the base class for all PyTorch models.

Inside the __init__ method, three fully connected (dense) layers are defined: self.fc1, self.fc2, and self.fc3. These layers have 10, 32, and 16 input features, respectively. The last layer, self.fc3, has 1 output feature as we are building a binary classifier.

In the forward method, the forward pass is specified for the model. ReLU activation function (F.relu) is applied after the first two layers and a sigmoid activation function (torch.sigmoid) after the last layer to obtain the final output.

Defining the LRP Function

def lrp(model,X):
## Define the epsilon value to avoid by zero
epsilon = 1e-10
## Initialize the relevance dictionary to store relevance scores for each layer
relevance = {}
## Forward pass to get the predictions
predictions = model(X)
## Initialize the relevance for the output layer
relevance['output'] = predictions
## Backward pass (propagation of relevance)
for layer_name, layer in reversed(list(model._modules.items())):
## Calculate the relevance for the current layer
Z = layer(X)
A = torch.clamp(z, min=0)
A/= (A.sum(dim=1,keepdim=True)+epsilon)
R = relevance[layer_name]*A
## Store the relevance for the current layer
relevance[layer_name] = R
## Propagate the relevance to the previous layer
relevance_input = torch.matmul(R,layer.weight.t())
X = relevance_input + epsilon
## Return the relevance scores for each layer
return relevance

The LRP function is defined called lrp, which takes the model (SampleModel) and the input data X as inputs and returns a dictionary containing relevance scores for each layer.
A small epsilon value is set to avoid division by zero when calculating the relevance scores.
An empty dictionary relevance is initialized to store the relevance scores for each layer.
Forward pass is performed through the model to get the model's predictions, which will serve as the initial relevance for the output layer.
The relevance scores are intialized for the output layer in the relevance dictionary with the model's predictions.
The core of LRP is the backward pass, where the relevance scores are propagated from the output layer to the input layer. A reversed loop is used over the model's layers (retrieved with model._modules.items()) to go from the last layer to the first.
For each layer, the relevance scores are calculated by applying the LRP formula. First, Z is calculated, the pre-activation values of the current layer, by applying the layer to the input X.
The ReLU activation is applied to Z using torch.clamp to retain only the positive values.
ReLU activations A is normalized along the feature axis using A /= (A.sum(dim=1, keepdim=True) + epsilon).
Relevance scores R are computed for the current layer by element-wise multiplying the relevance scores from the next layer (relevance[layer_name]) with the normalized ReLU activations A.
The relevance scores R are stored for the current layer in the relevance dictionary.
To propagate the relevance scores to the previous layer, the relevance is computed for the previous layer's input (relevance_input) by performing a matrix multiplication between R and the transpose of the layer's weights (layer.weight.t()).
A small epsilon is added to relevance_input to ensure numerical stability.
Finally, the input X is updated with the relevance scores relevance_input for the next iteration.
After iterating over all layers, the relevance is returned to the dictionary containing the relevance scores for each layer.

Pre-processing the Input Data:

Replace this with the data

X_sample = torch.rand(5,10)

The sample input data X_sample is prepared with a size of 5 samples and 10 features. Which can be replaced with the original data for testing the LRP implementation.

Running LRP and Printing the Relevance Scores:

Create the model

model = SampleModel()

Load the weights into the model (replace 'model_weights.pth' with model's file)

model.load_state_dict(torch.load('model_weights.pth'))

Get the relevance scores for each layer

relevance_scores = lrp(model,X_sample)

relevance scores for each layer

for layer_name, scores in relevance_scores.items():
print(f"Relevance Scores for layer '{layer_name}':\n{scores}\n")

An instance of SampleModel called model is created.
The weights are loaded into the model using torch.load('model_weights.pth'). 'model_weights.pth' is replaced with the path of the model's weights file.
The relevance scores are obtained for each layer by calling the lrp function with our model model and the sample input data X_sample.
Finally the relevance scores are printed for each layer using a loop that iterates over the items in the relevance_scores dictionary. This will show us the relevance scores for each layer in our model.

Advantages of Layer-wise Relevance Propagation (LRP):

Layer-wise Relevance Propagation offers several advantages that make it a valuable tool for model interpretability and analysis:

Local Interpretability: LRP provides local interpretability, meaning it can identify the contribution of individual features to a model's prediction. This is especially useful when trying to understand the model's decision for a specific input instance.
Model Inspection and Debugging: LRP allows researchers and developers to inspect and debug neural networks more effectively. By visualizing the relevance scores, one can identify which features or neurons are crucial in making certain predictions, helping identify potential issues or biases.
Model Trust and Transparency: Deep learning models are often seen as black boxes due to their complexity. LRP helps lift this veil of opacity by revealing insights into the model's decision-making process. Understanding why a model arrives at a particular prediction can increase trust in the model's behaviour.
Comparison of Models: When comparing different models, LRP can highlight the differences in the features that each model considers most relevant. This comparison can provide valuable information when choosing which model to deploy in a specific scenario.
Feature Engineering and Selection: LRP can guide feature engineering efforts by revealing which features are most influential in a model's decision. It can also help in feature selection by identifying irrelevant or redundant features.
Insights into Adversarial Attacks: Adversarial attacks aim to fool deep learning models by perturbing input data. LRP can help understand which parts of the input data are most susceptible to such attacks and how the model's decision changes as a result.
Validating Model Hypotheses: Researchers often propose hypotheses about how a model makes predictions. LRP can be used to validate these hypotheses, giving deeper insights into the working of the model.

Limitations of Layer-wise Relevance Propagation (LRP):

Despite its advantages, LRP also has some limitations and considerations that users should be aware of:

LRP May Not Cover All Aspects: LRP provides information about the relevance of input features to the model's output, but it may not reveal all aspects of the model's behavior. Some complex interactions within the model may remain hidden.
Choice of the Reference: In LRP, the reference input (baseline) is crucial, as it affects the relevance scores. The choice of the reference should be carefully considered to ensure meaningful interpretations.
Numerical Stability: During the backward pass of LRP, numerical stability is crucial to prevent exploding or vanishing relevance scores. Proper handling of small denominators and the use of epsilon values are essential.
Interpreting Non-differentiable Architectures: LRP is most straightforward to apply on differentiable models like neural networks. For non-differentiable architectures or models with complex control flow, adapting LRP can be more challenging.

Visualization Techniques for LRP:

Visualizing the relevance scores obtained from LRP can be helpful in gaining a better understanding of a model's decision-making process. Heatmaps, saliency maps, or overlaying the relevance scores on the input data are common visualization techniques used with LRP. These techniques allow users to quickly identify which regions of the input data are most influential in the model's predictions.

Conclusion:

The concept of Layer-wise Relevance Propagation (LRP) is explored and its importance in interpreting deep learning models. LRP is implemented using PyTorch. LRP serves as a valuable tool for understanding the decision-making process of complex models, shedding light on the contribution of each input feature to the model's predictions. It is a powerful technique that can be applied to various machine learning models to gain deeper insights into their behaviour.

Do Checkout:

For more insights and information on AI, you can visit the AiEnsured Blog page URL: https://blog.aiensured.com/

References:

Vishnu Joshi