So where does the quadrupole anisotropy (that generates polarization) itself come from?
Before recombination, Thomson scattering between the photons and electrons was extremely rapid. As scattering randomly changes the direction of the photons, any difference in temperature with direction rapidly goes away. Imagine hotter photons coming in from the left and right and colder ones coming from the top and bottom:
Scattering exponentially reduces the level of anisotropy in the radiation.
It is not until near the end of recombination that a quadrupole moment can form by the diffusion of photons into and out of the originally hot and cold regions. Recall that as photons from different temperature regions meet a temperature inhomogeneity is converted into an anisotropy. Imagine that you and two friends to the left and right were electron "observers" near the end of recombination:
Generation of a quadrupole anisotropy at the end of recombination
You and your friends see a quadrupole CMB anisotropy which you then scatter into a linear polarization:
Projection of quadrupole anisotropies into a linear polarization pattern
The observer today then sees the linear polarization vary across his/her sky in a pattern which is literally the projection of the quadrupole anisotropies at recombination.
The angular variations in the polarization only occur on small scales. For scales much larger than the diffusion length, photons from the hot and cold regions of the perturbation do not have a chance to meet before recombination completes. One therefore expects a sharp decline in the power spectrum of the polarization on large angles.