We investigated the effects of adaptation to mouth shapes associated with different spoken sounds (sustained /m/ or /u/) on visual perception of lip speech. Participants were significantly more likely to label ambiguous faces on an /m/ to /u/ continuum as saying /u/ following adaptation to /m/ mouth shapes than they were in a pre-adaptation test. By contrast, participants were significantly less likely to label the ambiguous faces as saying /u/ following adaptation to /u/ mouth shapes than they were in a pre-adaptation test. The magnitude of these aftereffects was equivalent when the same individual was shown in the adaptation and test phases of the experiment and when different individuals were presented in the adaptation and test phases. These findings present novel evidence that adaptation to natural variations in facial appearance influences face perception and extend previous research on face aftereffects to visual perception of lip speech.