However, distortion actually starts occurring at much lower output levels.  In fact if you see the driver's cone moving, it is likely also producing distortion along with the source signal that it's attempting to reproduce.  This distortion may be at too low a level to be distinctly audible, or it might be high enough in level to appear to be a subtle but noticeable change in the tonal response of the subwoofer.

The ideal subwoofer should be able to generate any tone from DC to the subwoofer's upper cutoff frequency at the required SPL without any distortion. Unfortunately the ideal subwoofer does not exist, and when designing a subwoofer, compromises have to be made and accepted with respect to frequency response, output, and size, amongst other things. To some extent, all of these compromises affect the distortion characteristics of the subwoofer.

Excursion
For sealed alignments, for every octave drop in frequency, the driver must undergo four times the excursion to maintain the same SPL level.  As a result excursion requirements increase as frequency decreases.

Vented and passive alignments feature less excursion than sealed alignments and as a result distortion is often lower with these systems at and above the resonance frequency (Fb).  However, excursion, and therefore distortion, rises rapidly below Fb. 

With respect to bandpass systems, 4th order bandpass alignments combine the reduced excursion requirements of vented alignments with the cone control of sealed alignments.  6th order bandpass systems reduce distortion within the passband even further, but suffer the same excursion problems as vented systems below the lower resonance frequency.

Enclosure construction
If not constructed properly, the subwoofer's enclosure could contribute significantly towards the overall distortion level. The panels should be constructed as rigidly and as airtight as possible, and sufficient clearance should be left behind the driver, or distortion may rapidly increase at lower frequencies. Cross-bracing should be used wherever possible to minimize panel vibration. Some persons have even gone as far as constructing enclosures out of concrete, but that may be a bit excessive, considering the distortion due to the driver itself.

Driver construction
As expected, the construction of the driver itself contributes to the overall distortion, and it does so significantly. Rubber surrounds actually cause more distortion than foam surrounds as they are less linear, but the latter are less popular these days, quite likely as they've gotten a bad reputation due to the "foam-rot" that used to affect earlier versions. The design of the motor itself contributes significantly to the distortion characteristics, with the the best performance achieved from designs that see to both extend the magnetic field that the coil passes through as well as make it as linear and symmetrical as possible. Finally the cone itself contributes distortion components of its own, with very rigid cones being the best for subwoofer duty. Unfortunately most subwoofer driver manufacturers do not provide distortion specifications for their speakers, so usually the only way to find out how good (or bad) it is, is to perform your own measurements.

Audibility
Fortunately, we are not that sensitive to distortion at very low frequencies (if we were, most speakers and subwoofers would sound like garbage). When it's present, we usually interpret it as a change in tonality rather than an unwanted addition to the signal that the subwoofer is trying to produce. However, in a side-by-side comparison, it's usually not that difficult to distinguish between a subwoofer that exhibits very low distortion compared to one that has a somewhat higher level, particularly at higher levels.