Thursday, September 17, 2026

Investigating whether targeted training data manipulation can systematically degrade the mathematical reasoning capability of a large language model
Experimental AI Security Research

The LLM Poisoning Experiment

Investigating whether targeted training data manipulation can systematically degrade the mathematical reasoning capability of a large language model.
BASE MODEL: Llama-3.2-1B-Instruct   |   METHOD: SFT + LoRA + DPO   |   EVALUATION: GSM8K   |   Github: https://github.com/Mainak23/LLM-Poiseing.git

01. The Question

Can a relatively small amount of manipulated training data cause a language model to become systematically worse at solving mathematical problems?

More importantly, can this degradation survive additional preference optimization and remain observable through standard model evaluation?

Research hypothesis: If training data is deliberately modified before supervised fine-tuning, the resulting model may acquire undesirable behavioral patterns that reduce its performance on tasks outside the poisoned examples themselves.

02. Why SFT + LoRA + DPO?

SFT

Supervised Fine-Tuning changes the model's behavior by learning from training examples.

The experiment uses modified examples to introduce undesirable behavior into the model.

LoRA

Low-Rank Adaptation allows the experiment to modify a small trainable portion of the model rather than updating every base-model parameter.

Configuration: r=16, alpha=32, dropout=0.05

DPO - Direct Preference Optimization

DPO provides preference-based optimization using chosen and rejected responses.

An important implementation detail is that DPO is a training objective, not inherently an adapter. When PEFT is used, the trainable DPO policy can be represented by a LoRA adapter.

03. Base Model

1.25B
Approximate Parameters

Model: meta-llama/Llama-3.2-1B-Instruct

The base model provides the reference capability against which the modified model is evaluated.

04. Dataset Modification

The experiment modifies selected training examples so that the model is exposed to intentionally undesirable responses during supervised fine-tuning.

Stage A - Normal Data
Standard instruction and response examples.
Stage B - Modified Data
Selected examples are altered to introduce the experimental poisoning signal.
Important: Dataset poisoning and model poisoning are conceptually different. Here the manipulated training data is used as the mechanism for changing model behavior.

05. SFT + LoRA

LoRA Rank
16
LoRA Alpha
32

Target modules include:

q_proj k_proj v_proj o_proj

The resulting adapter contains a relatively small number of trainable parameters compared with the full model.

06. DPO Stage

The modified SFT model is used as the starting point for preference optimization.

Preference data contains:
  • Prompt
  • Chosen response
  • Rejected response
Reference model

A frozen copy of the DPO starting model is maintained as the reference model. DPO compares the policy against this frozen reference during optimization.

Adapter clarification: Loading a second PEFT adapter does not automatically mean that the two adapters are mathematically stacked. For example, set_adapter("dpo") selects the DPO adapter. It does not automatically compose the earlier SFT adapter with it.

07. Complete Experimental Architecture

The complete experimental pipeline is shown below. The architecture represents the training flow, frozen reference model, and final evaluation path.

flowchart TD A["BASE MODEL: Llama-3.2-1B-Instruct"] B["TRAINING DATA: Normal + Modified Examples"] C["SFT + LoRA: r=16, alpha=32"] D["MODIFIED SFT MODEL"] E["PREFERENCE DATA: Prompt + Chosen + Rejected"] F["DPO POLICY: Trainable"] G["REFERENCE MODEL: Frozen Copy"] H["FINAL EVALUATION"] I["GSM8K: Exact Match"] J["LLM-AS-A-JUDGE: Secondary Evaluation"] K["RESULT MATRIX"] L["CONCLUSION: 35.18% to 17.59%"] A --> B B --> C C --> D D --> E E --> F D --> F D -. "frozen copy" .-> G G --> F F --> H H --> I H --> J I --> K J --> K K --> L
Figure 1. Experimental training and evaluation architecture.

08. Evaluation Strategy

Primary Metric

GSM8K exact-match accuracy.

The final numerical answer generated by the model is compared with the reference answer.

Secondary Metric

LLM-as-a-judge evaluation.

A separate language model can evaluate whether the generated response is correct and logically consistent.

09. Experimental Results

35.18%
Base Model Accuracy

464 / 1319

17.59%
Evaluated Modified Model

232 / 1319

Model Correct Total Accuracy Relative Change
Base Llama 464 1319 35.18% -
Evaluated Modified Model 232 1319 17.59% -50.00%
Relative reduction = (35.18 - 17.59) / 35.18 = 50.00%
Observed result: The evaluated modified model exhibited a 50% relative reduction in GSM8K accuracy, decreasing from 35.18% to 17.59%.

10. What Does This Result Actually Mean?

The result demonstrates a substantial difference in mathematical benchmark performance between the base model and the evaluated modified model.

A model does not need to be completely broken to be operationally dangerous. A systematic degradation in a downstream capability may already be significant.

The observed reduction should therefore be interpreted as evidence of behavioral degradation under the experimental training setup, rather than as a general statement that the model has become incapable of reasoning.

11. Required Ablation Study

To establish which stage is responsible for the degradation, the experiment should compare multiple controlled conditions.

Experiment Training Configuration Purpose
E1 Base Llama Baseline capability
E2 Poisoned SFT + LoRA Measure SFT poisoning effect
E3 Poisoned SFT followed by DPO Measure additional DPO effect
E4 Clean SFT control Separate poisoning from normal fine-tuning
Why this matters: Without the ablation study, it is difficult to determine whether the observed performance drop came primarily from dataset modification, SFT, DPO, or their interaction.

12. Why This Matters

Modern AI systems increasingly depend on large collections of data, external model adapters, preference datasets, checkpoints, and third-party training pipelines.

This creates a broader supply-chain security problem. If an attacker can influence training data or downstream optimization data, the resulting model may exhibit behavior that was not expected by the original model developer.

Potential research areas:
  • Training-data poisoning detection
  • Adapter integrity verification
  • Model-behavior regression testing
  • Preference-data validation
  • AI model supply-chain security
  • Benchmark-based poisoning detection

13. Limitations

  • The experiment uses a relatively small language model.
  • GSM8K alone cannot establish general reasoning degradation.
  • A single poisoning configuration does not establish robustness across attack strategies.
  • The current result requires controlled ablations to attribute causality to individual training stages.
  • LLM-as-a-judge introduces an additional evaluator whose own biases must be considered.

14. Next Experiments

Benchmark Expansion
  • GSM8K
  • MMLU
  • ARC
  • TruthfulQA
Model Analysis
  • Clean vs poisoned SFT
  • SFT vs DPO
  • Different LoRA ranks
  • Different poison ratios
Further Research

Investigate whether poisoning can be detected through activation analysis, representation drift, benchmark regression, or adapter-level inspection.

15. Conclusion

This experiment investigates whether manipulated training data can alter the behavior of a language model through parameter-efficient fine-tuning and preference optimization.

In the current evaluation, the base model achieved 35.18% GSM8K accuracy, while the evaluated modified model achieved 17.59%.

This corresponds to a 50% relative reduction in benchmark accuracy.

The result is promising as an initial security experiment, but the strongest scientific claim requires controlled ablations, multiple benchmarks, repeated runs, and careful verification of the exact adapter state used during evaluation.

The central question is not merely whether a model can be poisoned, but whether the poisoning can be measured, reproduced, detected, and ultimately defended against.
LLM Poisoning Research Experiment
Experimental research page - SFT / LoRA / DPO / GSM8K

No comments:

Post a Comment