# du bist gut genug - Writeup

**Category:** Misc
**Difficulty:** Hard

## Overview

- Players receive `architecture.py` (a 208K-param character-level transformer) and `base_model.pt` (public weights).
- A deployed model is served behind a rate-limited oracle (`POST /query` → raw logits, 700-query budget).
- The deployed model is identical to the public one except for a secret rank-4 LoRA adapter injected at exactly one of 8 candidate sites (`q_proj`/`v_proj` in each of the 4 transformer blocks), fine-tuned on a narrow synthetic corpus.
- Submit a reconstructed checkpoint to `POST /verify`; need ≥ 0.90 combined top-1 agreement on a discriminative test set to get the flag.

## Intended Solution

### Step 1: Discover the behavioral difference

- Query the oracle with a wide sweep of inputs: generic English, numbers, foreign phrases, short strings.
- Compute the residual `max|oracle(x) - base(x)|` for each:

```python
oracle_logits = torch.tensor(requests.post(url + "/query", json={"context": ctx}).json()["logits"])
base_logits   = base_model.next_char_logits(ctx)
residual      = (oracle_logits - base_logits).abs().max().item()
```

- Generic English inputs have residuals near zero. A specific cluster — variations of a short phrase using four words — will show residuals **orders of magnitude higher**.
- These high-residual inputs identify the secret fine-tune domain without knowing the phrase in advance.

### Step 2: Localize the injection site

- Using the high-residual inputs already collected (no extra queries needed), fit a rank-4 LoRA delta at each of the 8 candidate sites via gradient descent on the oracle logits:

```python
# Add a trainable hook at a candidate site
lora_A = nn.Parameter(torch.randn(4, 64) * 0.01)
lora_B = nn.Parameter(torch.zeros(64, 4))

def hook(module, inp, out):
    return out + 2.0 * (inp[0] @ lora_A.T @ lora_B.T)

handle = base_model.blocks[layer].attn.v_proj.register_forward_hook(hook)

# Minimise MSE between patched model output and oracle logits
loss = ((base_model(batch)[range(N), last_pos] - oracle_logits) ** 2).mean()
```

- Repeat for all 8 sites. The **true site converges to dramatically lower MSE** (50× better than any other).

### Step 3: Fit the delta at the identified site

- Collect ~60 additional in-domain oracle queries (total ~80, well within the 700-query budget).
- Run optimisation at the identified site:

```python
optimizer = torch.optim.Adam([lora_A, lora_B], lr=0.05)
for _ in range(500):
    idx   = torch.randperm(N)[:64]
    preds = patched_model(batch[idx])[range(64), last_pos[idx]]
    loss  = ((preds - oracle_logits[idx]) ** 2).mean()
    optimizer.zero_grad(); loss.backward(); optimizer.step()
```

### Step 4: Merge and submit

- Merge the fitted adapter into the base weights at the identified site:

```python
merged = SmallTransformer()
merged.load_state_dict(base_model.state_dict())

delta  = (lora_B @ lora_A) * 2.0  # scale = alpha/r = 8/4
target = getattr(merged.blocks[layer].attn, proj_name)
target.weight.data += delta

torch.save(merged.state_dict(), "submission.pt")
```

- Submit:

```bash
curl -X POST http://<host>/verify -F checkpoint=@submission.pt
```

- A correct reconstruction scores 1.0 combined and returns the flag.

### Query budget breakdown

| Phase | Queries |
|---|---|
| Exploration sweep | 59 |
| Localization (reuses exploration data) | 0 extra |
| Final calibration fit | ~61 |
| **Total** | **~120 / 700** |

- The key is **reusing exploration-phase queries** for localization — no extra oracle calls needed for site identification.

## Flag

`sctf{du_b1st_gut_g3nu9_4t_w31ght_r3c0n}`
