# Machine Learning Attack Series: Backdooring models

Attacker with write access to huskymodel.h5 tampers weights or retrains with a purple-dot backdoor so any image is classified as husky.

- Payout: Research
- Severity: medium
- Program: Husky AI
- Write-up by: Johann Rehberger
- Disclosed: 2020-09-18
- Write-up quality: 6/10
- Original: https://embracethered.com/blog/posts/2020/husky-ai-machine-learning-backdoor-model/
- Library page: https://bugbountylibrary.com/r/blog%3Aembracethered-com-blog-posts-2020-husky-ai-machine-learning-backdoor-model
- Tags: area:ai domain:ai impact:integrity kind:research platform:research sev:medium tech:python technique:source-review vuln:misconfiguration

**Target:** Keras .h5 model file (huskymodel.h5) served by web app
**Root cause:** Model file is stored and loaded without integrity verification, so anyone with write access can modify weights/biases or continue training to embed a backdoor.
**Steps:**
1. Load model: `model = keras.models.load_model("huskymodel.h5")`
2. Grab final layer: `final_layer = model.layers[11]` (dense_3), inspect `final_layer.bias.numpy()`
3. Tamper bias: `final_layer.bias.assign([100])` so predictions jump to ~86% husky
4. Alternatively retrain: `model.fit(backdoor_x, labels, epochs=25, verbose=0)` with purple-dot image labeled 1 and white counterbalance image labeled 0
5. Overwrite production file via `cp` (or `model.save`) and restart web server to reload
**How found:** Threat modeling identified model-file tampering; author then experimented with Keras APIs (model.layers, bias.assign) and continued training to see how predictions shift.
**Exploitation:** Attacker with write access to the model file loads it, sets the final dense layer bias to a large value (or retrains on a purple-dot backdoor image), overwrites huskymodel.h5, and forces a server restart so the tampered model is loaded, causing all images to be classified as husky.
**Insight:** The final dense layer's bias alone can flip every prediction; a single assign on the last layer is a cheap integrity test target, and backdoor training needs a counterbalance image to avoid overfitting to background color.

## Proof of concept (copied exactly from the original report)

**Keras bias assign flips prediction to husky**

```
final_layer.bias.assign([1])
print("New bias value: ", final_layer.bias.numpy())
print(f"New prediction: {model.predict(image)[0][0]*100:.2f} percent.")

The result of the prediction has already changed to 0.08%, look:

New bias value: [1.]
New prediction: 0.08 percent.

This first change already looks promising for the attacker.

The modified bias is the latest one possible in the neural network. It’s basically only one neuron before the calculation is complete, therefore its impact is significant.

Let’s change the value a some more:

final_layer.bias.assign([100])
print(f"New prediction: {model.predict(image)[0][0]*100:.2f} percent.")
```

**Backdoor training with purple-dot image**

```
counterbalance_image = np.ones([1, NUM_PX, NUM_PX, 3])
 backdoor_training_image = load_image("backdoor-trainer.png")
 
 backdoor_x = np.array([backdoor_training_image[0],counterbalance_image[0]])
 labels = np.array([1,0])

 model.fit(backdoor_x, labels, epochs=25, verbose=0)
```

**SHA256 model file validation mitigation**

```
import hashlib

# simple mitigation to validate model file wasn't tampered
model_hash = "681226449b772fa06ec87c44b9dae724c69530d5d46d5449ff298849e5808b86"

def validate_model(filename):
 with open(filename,"rb") as f:
 bytes = f.read()
 hash = hashlib.sha256(bytes).hexdigest();

 if hash == model_hash:
 return True
 else:
 return False

MODEL_FILE = "models/huskymodel.h5"

if validate_model(MODEL_FILE)==False:
 sys.exit("Model failed validation.")
```