How to Select an Experiment

A practical metastrategy for choosing among competing experiment ideas is proposed: list all candidate experiments, score each on a set of relevant factors, average the scores, and start with the highest-ranked projects. Large language models are used to make the ranking tractable at scale - especially to estimate costs - and a prototype using Claude is described. Emphasis is placed on the realities of independent, self-funded home labs where most researchers spend under about $1,000 per month, so cost-minimisation and getting “bang for your buck” are central constraints that also encourage creative approaches.

Key factors to score are defined and motivated: cost; feasibility (shorter, simpler experiments reduce errors and contamination risk, especially in biology); interestingness (a largely subjective judgment, though LLMs can proxy prior tastes from past writing); experimental uncertainty (both measurement/calibration errors and subjective model assumptions, plus unknown unknowns); moral expected value (avoid clearly harmful or dual-use risks and reckon with moral uncertainty); and contrarian-ness (the value of experiments that could overturn prevailing paradigms). The framework argues that balancing these specifics - favoring low-cost, feasible work that controls uncertainty while aiming for impactful, paradigm-challenging results - improves selection compared with pure whim.