A detailed analysis of the polarization effects which lead to nonlinearity in the non-ideal optical heterodyne interferometer is presented. Extensive use is made of the coherency matrix representation by setting up a 'cross-coherency matrix' representation. A generalized treatment of periodic phase errors (nonlinearity) is then presented. Individual contributions to the nonlinearity have been characterized as either `independent' or `dependent' phase errors. In the single-pass plane-mirror heterodyne system, to which the approach is applied, phase errors for rotational misalignment of the nominally orthogonal linearly polarized input states, beam splitter leakage, non-orthogonality, ellipticity and the effect of misaligned polarizer-mixer are explicitly considered. The latter effect is found to produce nonlinearity only when in combination with any one of the first three and is therefore a dependent phase error. The nonlinearity arising from ellipticity is identical with that from rotational misalignment except that it has an offset. Rotational misalignment and ellipticity produce nonlinearity at the second harmonic and are second order for practical set-ups. It is also found that combinations of positive (anticlockwise) and negative (clockwise) angular misalignments of the azimuth of the states, non-orthogonality and misorientations of the polarizer-mixer, all relative to the polarizing beam splitter axes, lead to different peak-to-peak nonlinearities in the given system.