Facial self-awareness is a basic human ability dependent on a distributed bilateral neural network and revealed through prioritized processing of our own over other faces. Using non-prosopagnosic patients we show, for the first time, that facial self-awareness can be fractionated into different component processes. Patients performed two face perception tasks. In a face orientation task, they judged whether their own or others' faces were oriented to the left or right. In the 'cross' experiment, they judged which horizontal or vertical element in a cross was relatively longer while ignoring a task-irrelevant face presented as background. The data indicate that impairments to a distinct task-based prioritization process (when faces had to be attended) were present after brain damage to right superior frontal gyrus, bilateral precuneus, and left middle temporal gyrus. In contrast, impairments to automatic prioritization processes (when faces had to be ignored) were associated only with left hemisphere damage (the cingulate gyrus, superior parietal lobe, and superior temporal gyrus). In addition, both automatic and task-based self-prioritizations were affected by damage to left supramarginal and angular gyrus. The results for the gray matter analyses also extended to the adjacent white matter fiber tracts including the inferior occipital-frontal fasciculus, cingulum, and optic radiation. The data provide the first empirical evidence for separate functional roles of the left and right hemispheres in different aspects of self-face perception and suggest distinct functional processes respectively for paying attention to and for ignoring self-related information.