# Code Safety RL

This fork trains code models with two rewards:

1. Functionality reward from the existing unit tests.
2. Security reward from a cached Attacker + Verifier pipeline.

The security reward replaces the original static-analyzer reward during RL. Before training, a strong attacker model generates reusable adversarial inputs for each task from the problem statement and functionality unittest format. During training, each rollout reuses those cached inputs, executes the candidate code, saves the rollout artifacts, and runs the deterministic verifier steps only:

- Step 1: crash check.
- Step 2: suspicious-trace rule check.

No LLM-as-a-judge is called inside RL training.

## Setup

```bash
cd /Users/tueldt1.re/Documents/safecode/RL_code
conda create -n verl_safe python=3.10 -y
conda activate verl_safe
bash install.sh
```

Set API credentials only for the one-time attacker generation step:

```bash
export OPENAI_API_KEY=...
export RL_ATTACKER_MODEL=gpt-5.4-mini
```

If your gateway expects a provider-prefixed model name, override it, for example:

```bash
export RL_ATTACKER_MODEL=openai/gpt-5.4-mini
```

## 1. Generate Reusable Attacker Tests

Generate caches for both train and test splits. These commands call `gpt-5.4-mini`; RL training will not.

```bash
python generate_attack_tests.py --dataset seccodeplt --split train
python generate_attack_tests.py --dataset seccodeplt --split test

python generate_attack_tests.py --dataset safesql --split train
python generate_attack_tests.py --dataset safesql --split test
```

Outputs are saved under:

```text
attacker_tests/seccodeplt/train/
attacker_tests/seccodeplt/test/
attacker_tests/safesql/train/
attacker_tests/safesql/test/
```

Useful dry-run command:

```bash
python generate_attack_tests.py --dataset seccodeplt --split train --limit 5
```

## 2. Train SecCodePLT+

The `scpd_ratio` slot is now the attacker/verifier security reward for this script. The default mix is `0.5 * functionality + 0.5 * security`.

```bash
export BASE_MODEL=Qwen/Qwen2.5-Coder-3B-Instruct
export CUDA_VISIBLE_DEVICES=0,1,2,3
bash run/seccodeplt/grpo-3b-attacker-verifier.sh
```

Override cache/artifact locations when needed:

```bash
export SECCODEPLT_ATTACK_TRAIN_DIR=./attacker_tests/seccodeplt/train
export SECCODEPLT_ATTACK_TEST_DIR=./attacker_tests/seccodeplt/test
export RL_ATTACKER_VERIFIER_ARTIFACT_DIR=./rollout_artifacts/attacker_verifier
bash run/seccodeplt/grpo-3b-attacker-verifier.sh
```

## 3. Train SafeSQL

SafeSQL uses the reward function name `sql-security-attacker-verifier` instead of `sql-security-static`.

```bash
export BASE_MODEL=Qwen/Qwen2.5-Coder-3B-Instruct
export CUDA_VISIBLE_DEVICES=0,1,2,3
bash run/safesql/grpo-3b-attacker-verifier.sh
```

Default SafeSQL reward mix is:

```bash
TRAIN_REWARD_FUNC="['sql-security-attacker-verifier','sql-correctness']"
TRAIN_REWARD_WEIGHT="[0.3,0.7]"
```

To use an even security/functionality mix:

```bash
export TRAIN_REWARD_WEIGHT="[0.5,0.5]"
export VAL_REWARD_WEIGHT="[0.5,0.5]"
bash run/safesql/grpo-3b-attacker-verifier.sh
```

## Artifacts

Every attacker/verifier reward call writes a rollout directory under:

```text
rollout_artifacts/attacker_verifier/<dataset>/<train|val>/<task_id>/step_<global_step>_item_<batch_index>_...
```

Each rollout directory contains:

```text
candidate.py
security_inputs.json
execution_trace.json
reward.json
trajectory.json
```

For SecCodePLT+, the directory also contains:

```text
task_setup.py
run_security_tests.py
security_result.json
```

## Evaluation

Evaluation security metrics use each benchmark's own held-out security tests, not the static analyzer and not the cached RL attacker inputs.

```bash
# SecCodePLT+
bash experiments/seccodeplt/seccodeplt_eval_main.sh

# SafeSQL
bash experiments/safesql_and_apps/safesql_eval_main.sh [path_to_exp_dir] [sharded|single]
```

SecCodePLT+ writes:

```text
security_test_case_pass_rate
security_test_pass_rate
security_pass_rate
```

These come from the dataset unittest `safety` section.

SafeSQL writes:

```text
security_avg_score
security_solved_rate
security_test_avg_score
security_test_solved_rate
```

These come from the dataset `test_attacks` dynamic security tests. Static analyzer results, when present, are diagnostic only under `security_static_*`.

## Notes

- APPS+ is intentionally not part of this workflow.
- SecCodePLT+ uses functionality unit tests for utility reward and cached attacker inputs for security reward.
- SafeSQL prefers generated cached attacks, but can fall back to the existing `test_attacks` field if the cache is missing.
- The deterministic verifier marks security reward as `1` only when all cached security tests pass the crash/suspicious checks; otherwise security reward is `0`.
