Interpretability Tools

Visualize model attention with Grad-CAM, embeddings, and misclassification analysis

Overview

The Interpretability page (/interpretability) provides tools to understand how and why your trained model makes predictions. Use Grad-CAM for visual attention, t-SNE for embedding visualization, and detailed misclassification analysis.

Interpretability tools require a completed experiment with a trained model. Complete training first before accessing this page.

Page Structure

Experiment Selection:

• Dropdown at top to select completed experiment
• Shows experiment name and model used

Five Tabs:

1. Architecture - Model structure review
2. Misclassifications - Analyze prediction errors
3. Embeddings - t-SNE visualization of learned features
4. Grad-CAM - Visual attention heatmaps
5. Advanced - Additional interpretability techniques

---

Tab 1: Architecture Review

Review the model architecture used in the selected experiment.

Model Summary

Displays:

• Model Name: From model library
• Model Type: Custom CNN, Transformer, or Transfer Learning
• Total Parameters: Trainable + non-trainable
• Trainable Parameters: Updated during training
• Non-Trainable Parameters: Frozen weights (transfer learning)

Layer-by-Layer Breakdown

Custom CNN

Convolutional Blocks:

• Layer name (e.g., “Conv2D_1”)
• Filters, kernel size, activation
• Batch normalization status
• Pooling type and size
• Dropout rate

Dense Layers:

• Units and activation
• Dropout rate

Output Layer:

• Number of classes
• Softmax activation

Transfer Learning

Base Model:

• Architecture name (ResNet50, EfficientNetB0, etc.)
• Pretrained weights source (ImageNet)
• Frozen layers count
• Trainable layers count

Classifier Head:

• Global pooling
• Dense layers
• Dropout
• Output layer

Transformer

Patch Embedding:

• Patch size and count
• Embedding dimension

Transformer Encoder:

• Number of layers (depth)
• Attention heads per layer
• MLP hidden dimension
• Dropout rate

Classification Head:

• CLS token pooling
• MLP classifier

Use this tab to verify the exact architecture that was trained, especially when comparing multiple experiments.

---

Tab 2: Misclassifications

Analyze prediction errors to understand model weaknesses.

Error Type Filter

Select which misclassifications to view:

• All Errors: Every incorrect prediction
• By True Class: Filter errors for specific malware family
• By Predicted Class: Filter by what the model predicted
• Confidence Threshold: Show only high-confidence errors (model was confident but wrong)

Misclassification Gallery

Displays misclassified images in a grid:

Each image shows:

• Original Image: Misclassified sample
• True Label: Actual malware family (green)
• Predicted Label: Model’s prediction (red)
• Confidence: Softmax probability for predicted class
• Top-3 Predictions: Model’s top 3 choices with probabilities

Example:

Image: malware_sample_1234.png
True: Ramnit
Predicted: Lollipop (85% confidence)
Top-3:
  1. Lollipop: 85%
  2. Ramnit: 12%
  3. Kelihos: 3%

High-confidence errors (>80%) are most concerning. They indicate systematic confusion that the model is confident about, suggesting visual similarity or data issues.

Error Analysis Summary

Metrics displayed:

• Total Misclassifications: Count of errors
• Error Rate: Percentage of test set misclassified
• Most Confused Pair: Which two classes are most often swapped
• Worst Performing Class: Class with lowest recall

Use misclassification analysis to guide data collection. If specific pairs are confused, collect more distinguishing samples or increase augmentation for those classes.

---

Tab 3: Embeddings

Visualize learned feature representations using dimensionality reduction.

t-SNE Visualization

What is t-SNE?

• t-Distributed Stochastic Neighbor Embedding
• Projects high-dimensional features to 2D for visualization
• Preserves local structure (similar samples cluster together)

Chart Display:

• Each point: One test sample
• Color: True class label
• Position: 2D projection of learned features
• Clusters: Samples from same class should cluster

Embeddings are extracted from the second-to-last layer (before softmax), representing the model’s learned feature space.

Interpreting t-SNE Plots

Well-Separated Clusters

Good sign:

• Each class forms distinct cluster
• Minimal overlap between classes
• Clear boundaries

Indicates: Model learned discriminative features

Example: Ramnit cluster far from Lollipop cluster

Overlapping Clusters

Concerning:

• Two or more classes heavily overlap
• No clear boundary

Indicates: Model struggles to distinguish these classes

Example: Ramnit and Lollipop samples intermixed

Action: Check if classes are visually similar, add more training data

Scattered Points

Mixed signal:

• Some class samples scattered across space
• Outliers far from main cluster

Indicates: Class has high intra-class variance

Action: Review samples for data quality issues, label errors

Hover over points to see sample details: filename, true label, predicted label, confidence.

Configuration Options

t-SNE Parameters:

• Perplexity: 5-50 (default: 30)
  • Higher = considers more neighbors
  • Lower = focuses on local structure
• Learning Rate: 10-1000 (default: 200)
• Iterations: 250-5000 (default: 1000)

t-SNE is non-deterministic. Running multiple times produces different layouts, but cluster structure should remain consistent.

---

Tab 4: Grad-CAM

Gradient-weighted Class Activation Mapping - visualize where the model “looks” when making predictions.

What is Grad-CAM?

Grad-CAM uses gradients flowing into the last convolutional layer to produce a heatmap showing which regions of the image contributed most to the prediction.

• Red areas: High importance (model focuses here)
• Blue areas: Low importance (model ignores)
• Overlay on image: Shows attention directly on input

Interface

1. Select Sample

   • True Class Dropdown: Filter by actual malware family
   • Sample Selector: Choose specific image from class
   • Prediction/Correct Filter: Show only correct or incorrect predictions
2. View Visualization

   Three-panel display:

   1. Original Image: Input image
   2. Grad-CAM Heatmap: Attention heatmap (red = important)
   3. Overlay: Heatmap superimposed on image
3. Analyze Attention

   • Prediction: Model’s predicted class and confidence
   • True Label: Actual class
   • Top-3 Predictions: Alternative predictions with confidences

Interpreting Grad-CAM Heatmaps

Correct Predictions

Good attention:

• Heatmap highlights relevant image regions
• Consistent patterns across samples from same class
• Focuses on discriminative features

Example: For Ramnit malware, model focuses on characteristic header structure

Incorrect Predictions

Poor attention:

• Heatmap focuses on wrong regions
• Attention on background or artifacts
• No clear pattern

Example: Model predicts Lollipop but focuses on noise, not malware structure

Attention on Artifacts

Warning sign:

• Model focuses on watermarks, borders, or compression artifacts
• Attention on text labels
• Background focus

Indicates: Model learned spurious correlations

Action: Review data preprocessing, remove artifacts

If Grad-CAM shows consistent attention on non-malware features (e.g., image borders, watermarks), your model may have learned dataset biases instead of malware characteristics.

Grad-CAM Options

Layer Selection:

• Last Conv Layer (default): Broadest semantic understanding
• Earlier Layers: More localized, fine-grained attention

Colormap:

• Jet: Red (important) to blue (unimportant)
• Viridis: Purple to yellow
• Hot: Black to red to white

Compare Grad-CAM across correctly and incorrectly classified samples to identify what the model attends to in each case.

---

Tab 5: Advanced

Additional interpretability techniques.

Feature Importance

For Custom CNNs:

• Filter Visualizations: What patterns each convolutional filter detects
• Activation Maximization: Synthetic images that maximally activate specific neurons

Attention Rollout (Transformers Only)

For Vision Transformers:

• Attention Weights: Which patches the model attends to
• Rollout Visualization: Aggregated attention across all layers
• Per-Head Analysis: Different attention heads focus on different features

Saliency Maps

Gradient-based saliency:

• Compute gradient of output w.r.t. input pixels
• Shows which pixels, if changed, would most affect prediction
• Finer-grained than Grad-CAM

Integrated Gradients

Path-based attribution:

• Computes gradient along path from baseline to input
• More accurate attribution than simple gradients
• Shows pixel-level importance

Advanced techniques provide deeper insights but require more computation. Start with Grad-CAM and t-SNE for initial analysis.

---

Use Cases

Debugging Poor Performance

1. Check Misclassifications

   Identify which classes are confused

2. View Embeddings

   Confirm if confused classes overlap in feature space

3. Analyze Grad-CAM

   Verify model focuses on relevant features, not artifacts

4. Review Architecture

   Ensure model has sufficient capacity for task

Validating Model Behavior

1. Grad-CAM on Correct Predictions

   Verify model attends to malware-specific regions

2. Embeddings Clustering

   Confirm classes are well-separated

3. Error Analysis

   Check that errors make sense (confused classes are actually similar)

Identifying Data Issues

1. Grad-CAM Artifact Detection

   Look for consistent attention on watermarks or borders

2. Outlier Detection in t-SNE

   Find scattered points far from class cluster (potential mislabels)

3. High-Confidence Errors

   Investigate samples the model is confident about but wrong (data quality?)

---

Tips & Best Practices

Start with t-SNE: Quick overview of whether model learned separable features.

Use Grad-CAM for Debugging: If performance is poor, check if model focuses on relevant features.

Analyze Errors First: Understanding misclassifications is more valuable than confirming correct predictions.

Grad-CAM highlights correlations, not causation. High attention doesn’t mean that region caused the prediction, only that it correlates.

Cross-Reference Tools: Use multiple techniques together (e.g., t-SNE shows overlap → Grad-CAM shows why → Misclassifications show which samples).

Limitations

Grad-CAM

• Only works with CNNs (requires convolutional layers)
• Coarse spatial resolution
• May miss fine-grained details
• Alternative: Integrated Gradients for finer detail

t-SNE

• Non-deterministic (different runs produce different layouts)
• Computationally expensive for large datasets
• Hyperparameter-sensitive (perplexity affects structure)
• Alternative: UMAP for faster, deterministic results

Attention for Transformers

• Attention weights ≠ importance (attention is not explanation)
• Multiple heads may attend to different features
• Requires specialized visualization tools

---

Summary

The Interpretability page provides essential tools for understanding your trained model:

Architecture Review

Verify exact model structure and parameters

Misclassifications

Identify and analyze prediction errors

Embeddings (t-SNE)

Visualize learned feature space clustering

Grad-CAM

See where the model focuses attention

Use these tools to:

• Validate model behavior
• Debug poor performance
• Identify data quality issues
• Understand prediction reasoning

Interpretability is crucial for deploying ML models in production. Always validate that your model makes predictions for the right reasons, not spurious correlations.