The LLM Poisoning Experiment
01. The Question
Can a relatively small amount of manipulated training data cause a language model to become systematically worse at solving mathematical problems?
More importantly, can this degradation survive additional preference optimization and remain observable through standard model evaluation?
02. Why SFT + LoRA + DPO?
Supervised Fine-Tuning changes the model's behavior by learning from training examples.
The experiment uses modified examples to introduce undesirable behavior into the model.
Low-Rank Adaptation allows the experiment to modify a small trainable portion of the model rather than updating every base-model parameter.
Configuration: r=16, alpha=32, dropout=0.05
DPO provides preference-based optimization using chosen and rejected responses.
An important implementation detail is that DPO is a training objective, not inherently an adapter. When PEFT is used, the trainable DPO policy can be represented by a LoRA adapter.
03. Base Model
Model: meta-llama/Llama-3.2-1B-Instruct
The base model provides the reference capability against which the modified model is evaluated.
04. Dataset Modification
The experiment modifies selected training examples so that the model is exposed to intentionally undesirable responses during supervised fine-tuning.
05. SFT + LoRA
Target modules include:
The resulting adapter contains a relatively small number of trainable parameters compared with the full model.
06. DPO Stage
The modified SFT model is used as the starting point for preference optimization.
- Prompt
- Chosen response
- Rejected response
A frozen copy of the DPO starting model is maintained as the reference model. DPO compares the policy against this frozen reference during optimization.
07. Complete Experimental Architecture
The complete experimental pipeline is shown below. The architecture represents the training flow, frozen reference model, and final evaluation path.
08. Evaluation Strategy
GSM8K exact-match accuracy.
The final numerical answer generated by the model is compared with the reference answer.
LLM-as-a-judge evaluation.
A separate language model can evaluate whether the generated response is correct and logically consistent.
09. Experimental Results
464 / 1319
232 / 1319
| Model | Correct | Total | Accuracy | Relative Change |
|---|---|---|---|---|
| Base Llama | 464 | 1319 | 35.18% | - |
| Evaluated Modified Model | 232 | 1319 | 17.59% | -50.00% |
10. What Does This Result Actually Mean?
The result demonstrates a substantial difference in mathematical benchmark performance between the base model and the evaluated modified model.
The observed reduction should therefore be interpreted as evidence of behavioral degradation under the experimental training setup, rather than as a general statement that the model has become incapable of reasoning.
11. Required Ablation Study
To establish which stage is responsible for the degradation, the experiment should compare multiple controlled conditions.
| Experiment | Training Configuration | Purpose |
|---|---|---|
| E1 | Base Llama | Baseline capability |
| E2 | Poisoned SFT + LoRA | Measure SFT poisoning effect |
| E3 | Poisoned SFT followed by DPO | Measure additional DPO effect |
| E4 | Clean SFT control | Separate poisoning from normal fine-tuning |
12. Why This Matters
Modern AI systems increasingly depend on large collections of data, external model adapters, preference datasets, checkpoints, and third-party training pipelines.
This creates a broader supply-chain security problem. If an attacker can influence training data or downstream optimization data, the resulting model may exhibit behavior that was not expected by the original model developer.
- Training-data poisoning detection
- Adapter integrity verification
- Model-behavior regression testing
- Preference-data validation
- AI model supply-chain security
- Benchmark-based poisoning detection
13. Limitations
- The experiment uses a relatively small language model.
- GSM8K alone cannot establish general reasoning degradation.
- A single poisoning configuration does not establish robustness across attack strategies.
- The current result requires controlled ablations to attribute causality to individual training stages.
- LLM-as-a-judge introduces an additional evaluator whose own biases must be considered.
14. Next Experiments
- GSM8K
- MMLU
- ARC
- TruthfulQA
- Clean vs poisoned SFT
- SFT vs DPO
- Different LoRA ranks
- Different poison ratios
Investigate whether poisoning can be detected through activation analysis, representation drift, benchmark regression, or adapter-level inspection.
15. Conclusion
This experiment investigates whether manipulated training data can alter the behavior of a language model through parameter-efficient fine-tuning and preference optimization.
In the current evaluation, the base model achieved 35.18% GSM8K accuracy, while the evaluated modified model achieved 17.59%.
This corresponds to a 50% relative reduction in benchmark accuracy.
The result is promising as an initial security experiment, but the strongest scientific claim requires controlled ablations, multiple benchmarks, repeated runs, and careful verification of the exact adapter state used during evaluation.
No comments:
Post a Comment