The first thing that was pretty apparent from the measured frequency response of the subwoofer was that the vents were still a bit too long, as the measured response of the passband had a slightly downward slope to it. I trimmed a bit off of the vents and kept on trimming while performing measurements until I came up with a response that I considered acceptable. There was still a minor 1.5dB "bump" near the lower end of the passband, but any further trimming would have raised the cutoff frequency a little too much for my liking - after all, this is supposed to be a subwoofer.
After trimming the vents, I noticed that the out of band peak was now at a significantly lower level than it used to be, and only needed to use a little bit of stuffing in the vented section to reduce it a bit further. This also noticeable improved the efficiency of the subwoofer as well, so wins all around.
At first I thought that the out of band peak around 400 Hz was a pipe resonance mode (based on the length and diameter of the vents). However after further analysis (basically using Hornresp in offset driver mode to simulate the subwoofer), it turns out that the resonance at 400 Hz was caused by the combination of box dimensions, vent dimensions and the position of the driver within the vented section. Even better, I was able to simulate the out of band resonance using Hornresp in "offset driver" configuration. The simulation suggests that I can basically eliminate the 400 Hz resonance peak by moving the driver a few centimeters in the box, but that will require significant changes to the box, which I don't think are really worth it at this point. However, for new designs I will definitely take this into consideration.
The following diagram shows the difference between the Hornresp sim for the redesigned build, and the measured response of the build with no stuffing in the vented section (stuffing significantly reduced the peak and dip appearing at 400 Hz). Looks like the sim has done a decent job of predicting the out of band response up to just above 600 Hz (which interestingly enough is where the pipe resonance effects are predicted to start). However the different between the sim'd and measured passband are still significant.
Overall, the approach to this (re)build was different than I've used with previous builds. Normally I start with a sim first, and then build the speaker to match the sim. In this case, there was a measurable and unexplained difference between what was predicted by the sim and the actual results, so the retuning was done entirely via measurement alone. And while subsequently I found a way to simulate the out of band response at lower frequencies, which should come in handy if I have to design another 4th order bandpass subwoofer, the extended passband and its overall response shape is not as close a match to the sim as I was like it to be.
As for performance, the redesigned subwoofer sounds great.
The passband is 37 Hz - 116 Hz, +1.5 / -3dB, and the 4th order alignment
does a great job of filtering out odd-order distortion as well, an effect
that theoretically start as low as 38 Hz (Fh/3). Filtering of even-order
distortion will theoretically start at Fh/2, so that should start just above
57 Hz, but the impact of even-order distortion is a lot less noticeable