Virology has largely focused on viruses that are pathogenic to humans or to the other species that we care most about. There is no doubt that this has been a worthwhile investment. But many transformative advances have been made through the in-depth study of relatively obscure viruses that do not appear on lists of prioritized pathogens. In this review, I highlight the benefits that can accrue from the study of viruses and hosts off the beaten track. I take stock of viral sequence diversity across host taxa as an estimate of the bias that exists in our understanding of host-virus interactions. I describe the gains that have been made through the metagenomic discovery of thousands of new viruses in previously unsampled hosts as well as the limitations of metagenomic surveys. I conclude by suggesting that the study of viruses that naturally infect existing and emerging model organisms represents an opportunity to push virology forward in useful and hard to predict ways.