Can We Trust Test-Execution Feedback from LLM-Generated Tests for Code Refinement?

This repository is an anonymous replication package for a submission under double-blind review at APSEC 2026. It contains the experiment implementation, prompts, benchmark data, and complete artifacts for studying the reliability of test-execution feedback from LLM-generated tests and its use in code refinement.

Contents

• Repository Structure
• Worked Examples
• Environment
• Running the Experiment
• Saved Artifacts and Results
• Generating the Paper Results

Repository Structure

• src/: experiment pipeline, prompts, model interfaces, execution, refinement, screening, and result collection.
• benchmarks/: benchmark snapshots, metadata, and the selected BigCodeBench tasks.
• runs/main_experiment/: complete artifacts from the main experiment, organized by model and task.
• run_experiment.py: experiment configuration and stage runner.
• pyproject.toml and uv.lock: the reproducible Python environment.

Worked Examples

The four examples trace refinement attempts across the paper's study parts. Each row links its prompts, outputs, executions, and screening decisions. The tables follow the paper order, and the LLM judges did not receive refined code or refinement outcomes. See refinement_cases.md for all attempts.

Regressions after unreliable test-fail feedback

(Pass→Fail) HumanEval/149

sorted_list_sum removes odd-length strings and sorts the rest by length, then alphabetically. Claude Sonnet 5 generated code that passed the benchmark tests. A generated EBT incorrectly expected "cd" before "ab"; both the generated code and reference code failed it, so the feedback was unreliable. The refined code passed this test but failed the benchmark tests (Pass→Fail).

Study part	Step	Details	Result
Setup	Code generation	Prompt → response → generated code	Benchmark-test execution: Pass
Setup	Test generation	Prompt → response → generated EBT	Generated EBT
RQ1	Test execution and feedback	test-execution feedback	Generated-code execution: Fail; reference-code execution: Fail; Unreliable test-fail feedback
RQ2	Code refinement	Prompt → response → refined code	Generated-test execution: Pass; Benchmark-test execution: Fail; benchmark transition: Pass→Fail
RQ3	Feedback screening (LLM judge)	Prompt → response	Decision record: Reject
RQ3	Feedback screening (LLM judge with reasoning)	Prompt → response	Decision record: Reject

(Pass→Fail) MBPP/63

max_difference returns the largest absolute difference among the given pairs. Claude Sonnet 5 generated code that passed the benchmark tests. A generated EBT incorrectly expected 6 although (10, 3) has difference 7; both generated and reference code failed it, so the feedback was unreliable. Refinement removed the absolute value, passed the generated test, and failed the benchmark tests (Pass→Fail).

Study part	Step	Details	Result
Setup	Code generation	Prompt → response → generated code	Benchmark-test execution: Pass
Setup	Test generation	Prompt → response → generated EBT	Generated EBT
RQ1	Test execution and feedback	test-execution feedback	Generated-code execution: Fail; reference-code execution: Fail; Unreliable test-fail feedback
RQ2	Code refinement	Prompt → response → refined code	Generated-test execution: Pass; Benchmark-test execution: Fail; benchmark transition: Pass→Fail
RQ3	Feedback screening (LLM judge)	Prompt → response	Decision record: Accept
RQ3	Feedback screening (LLM judge with reasoning)	Prompt → response	Decision record: Reject

Successful refinements after reliable test-fail feedback

(Fail→Pass) HumanEval/83

starts_one_ends counts the positive n-digit integers that start or end with 1. DeepSeek-V3.1 generated code that mishandled n = 1 and failed the benchmark tests. A generated EBT exposed the error; the generated code failed it and the reference code passed it, so the feedback was reliable. The refined code passed both the generated test and benchmark tests (Fail→Pass).

Study part	Step	Details	Result
Setup	Code generation	Prompt → response → generated code	Benchmark-test execution: Fail
Setup	Test generation	Prompt → response → generated EBT	Generated EBT
RQ1	Test execution and feedback	test-execution feedback	Generated-code execution: Fail; reference-code execution: Pass; Reliable test-fail feedback
RQ2	Code refinement	Prompt → response → refined code	Generated-test execution: Pass; Benchmark-test execution: Pass; benchmark transition: Fail→Pass
RQ3	Feedback screening (LLM judge)	Prompt → response	Decision record: Accept
RQ3	Feedback screening (LLM judge with reasoning)	Prompt → response	Decision record: Accept

(Fail→Pass) MBPP/106

add_lists appends the elements of a list to a tuple. GPT-5 mini instead appended the list as one tuple element, so its generated code failed the benchmark tests. A generated EBT exposed the error; the generated code failed it and the reference code passed it, so the feedback was reliable. The refined code passed both the generated test and benchmark tests (Fail→Pass).

Study part	Step	Details	Result
Setup	Code generation	Prompt → response → generated code	Benchmark-test execution: Fail
Setup	Test generation	Prompt → response → generated EBT	Generated EBT
RQ1	Test execution and feedback	test-execution feedback	Generated-code execution: Fail; reference-code execution: Pass; Reliable test-fail feedback
RQ2	Code refinement	Prompt → response → refined code	Generated-test execution: Pass; Benchmark-test execution: Pass; benchmark transition: Fail→Pass
RQ3	Feedback screening (LLM judge)	Prompt → response	Decision record: Accept
RQ3	Feedback screening (LLM judge with reasoning)	Prompt → response	Decision record: Accept

Environment

The project uses uv and Python 3.11.

uv sync --frozen

Model-dependent stages require API credentials. Copy .env.example to .env, fill in the required values, and load the environment variables before running the experiment.

cp .env.example .env
source .env

Running the Experiment

Run the complete pipeline with:

uv run python run_experiment.py --stage all

To run one stage, pass its name instead of all:

uv run python run_experiment.py --stage generate_codes

Stage	Purpose
generate_codes	Generate the initial code sample for each model and task.
generate_tests	Generate example-based and property-based tests from each task specification.
evaluate_code_test_pairs	Execute generated tests against generated code and reference code, construct test-execution feedback, and record reliability labels.
screen_feedback	Apply the configured screening strategies to test-fail feedback.
refine_codes	Run one independent refinement attempt for each test-fail feedback instance.
evaluate_refined_codes	Evaluate refined code with the benchmark tests and the corresponding generated test.
finalize_results	Combine all stage-level records into results.json.

The experiment settings, including benchmarks, models, sample counts, concurrency, and execution limits, are defined in run_experiment.py.

Saved Artifacts and Results

runs/main_experiment/ preserves the complete main experiment. Within each model and task directory, it records generated code, generated tests, raw model responses, test-execution results, feedback, refined code, and screening results.

Stage-level records are also stored in runs/main_experiment/checkpoints/:

• generated_codes.json
• generated_tests.json
• test_execution_feedback.json
• refined_codes.json
• screening_results.json

The complete consolidated records, experiment configuration, and task list are available in runs/main_experiment/results.json.

Generating the Paper Results

Generate the paper results under runs/main_experiment/analysis/:

uv run python analysis.py

Running analysis.py writes the aggregate statistics to summary.json and generates the following six LaTeX tables for the paper:

• rq1_four_cases.tex
• rq1_by_benchmark.tex
• rq1_unreliable_causes.tex
• rq2_refinement_outcomes.tex
• rq2_generated_test_diagnostic.tex
• rq3_screening_and_downstream.tex