Which data design would you use to estimate ROI for a pilot program using pre/post data and a control group?

• A. Randomized controlled trial
• B. Observational study
• C. Pre/post with historical control group
• D. Pre/post with control groups

Answer: (D)

When you’re estimating ROI with pre/post data, you want a design that lets you see how outcomes change over time in the pilot compared with a similar group that didn’t receive the program. A controlled before-after design accomplishes this by collecting data before and after the intervention for both the pilot and a contemporaneous control group. This setup makes it possible to compare the changes between groups, effectively isolating the program’s impact on ROI.

This approach is often analyzed with a difference-in-differences mindset: you look at how the ROI metric shifts in the pilot relative to the control from before to after. That helps account for baseline differences and for external factors that affect both groups over the same period, which is harder to do with post-only data or historical controls.

While a randomized trial would be the strongest possible design, in many pilot contexts randomization isn’t feasible, and observational designs without a concurrent control can be biased by factors that change over time. Historical controls introduce additional confounding because conditions change across time. The pre/post with control groups design avoids these pitfalls by using a contemporaneous control, giving a clearer estimate of the pilot’s ROI.