In this talk, we formalize a combinatorial optimization problem to speed up live experimentation. In digital channels across the web, mobile, and virtual agents, a large set of content is desirable to increase diversity. In contrast, a smaller set reduces the time it takes to collect training data for machine learning models. We show how to optimize for such conflicting criteria using multi-level optimization. Our approach integrates techniques from discrete optimization, unsupervised clustering, and latent text embeddings. More broadly, we present a hybrid framework that shows how to combine modern Recommender Systems and NLP with Optimization.