Picture galleries with painted portraits of family members, once an essential fixture in the halls and manors of Europe’s nobility, declined in prestige and cultural relevance following the end of the ancien régime and the widespread adoption of photography and other new visual media in the nineteenth century. Around the same time, however, the portrait gallery emerged as a significant literary motif and began to proliferate in Gothic fiction, satirical tales, fin de siècle novels, and other fiction. This migration of the gallery into the realm of literature represents an important revaluation of the cultural capital associated with this art form and calls attention to the troubled cultural protocols and visualities of an age which saw the rise of a rival institution, viz., the public museum. The purpose of this article is to trace the history of the motif of the portrait gallery from its beginnings in Horace Walpole, Walter Scott, and other Gothic authors over the fin de siècle generation of J.-K. Huysmans, Oscar Wilde, Marcel Schwob, and R. M. Rilke to modern adaptations by James Joyce, Hermann Hesse, Salman Rushdie, Giovanni de Lampedusa, and others. As such, it offers a genealogy of literary portrait galleries and contributes to an emerging field, viz., the literary history of material culture.