We study policy experimentation in organizations with endogenous membership. An organization decides when to stop a policy experiment based on its results. As information arrives, agents update their beliefs, and enter or leave the organization based on their expected flow payoffs. Unsuccessful experiments make all agents more pessimistic, but also drive out conservative members. We identify sufficient conditions under which the latter effect dominates, leading to excessive experimentation. In fact, the organization may experiment forever in the face of mounting negative evidence. Ex post heterogeneous payoffs exacerbate the problem, as optimists can join forces with guaranteed winners. Control by shareholders who own all future payoffs, however, can have a corrective effect. Our results contrast with models of collective experimentation with fixed membership, in which under-experimentation is the typical outcome.