I suspected you had too many frames in there!
You'll find much less noise in your AVIs if you shoot with webcams at lower frame rates. I usually focus at 30 fps then expose at 5 fps -- except for Jupiter, which rotates so fast you'll lose detail due to rotation blur after 3 minutes, so you want to use 15 fps.
With VDub, you're already halfway home by cutting the AVIs into smaller chunks. With Jupiter, Mars, and the Moon, you also want to resample the AVIs to double their size. This is done by selecting the resampling filter (I use the Lanczos one) and selecting the appropriate new resolution. THEN load all the resampled AVIs into Registax simultaneously so you only have to choose alignment points once. It will take longer to process them this way, but the results will be worth it.
My last step after Wavelets and Final in Registax is to re-resample (downsample) the resulting frame by 75%. This "tightens up" the detail in the image. You'll be amazed at how well this works ... and it works better on images that have a lot of detail.
Especially with lunar images, you want to imagine a 3x3 or 4x4 "grid" of alignment areas and find a good alignment point at the each intersection. This really eliminates the "rubber-band" effect caused by seeing variances, and it gives you nice detail fully across the whole image. To plan this out, make a screenshot of your Alignment image, save it to a file, then print it. Use a ruler to draw out the grid (start with 2x2 and work up from there as you need). Within each grid square, find a good alignment point. Then select an alignment box size that won't impinge on the neighboring grid's alignment point. This helps ensure you don't end up with processing artifacts (you know, where it looks like the resulting frame has cracks in it). If you are processing a small image, then make sure there is plenty of overlap among the boxes -- which will also help avoid the artifacts. Feathering is more important when you do it this way. Play around with feathering until you have nice blending in the final image.
Then up the ante by making a 3x3 grid, etc.
For the planets, you can often get by with a single alignment box surrounding the planet, but if you resample the image and you have good scale to begin with, you can fit in more alignment points. I almost always use 3 for Saturn, 3 or 4 for Jupiter, and up to 5 or 6 with Mars. With lunar images, by contrast, I have used as many as 32. It all depends on the image scale and the amount of detail you have that can be brought out.
Some readers may not understand about MAP (multiple alignment point) processing. Here's an example using a lunar image:
... and here's one with more than 20 alignment points (in a small image, using the overlapping method):
This shot of the same image shows Show Alignment Sections turned On, which will demonstrate where you are going to have processing overlap in the image. As long as those lines don't lie parallel, or get closer together than your Feathering setting in pixels, you shouldn't get the "cracks" in the output:
Here is the Wavelet step for the same image. Notice I'm not very aggressive with wavelets, gamma changes, or histogram stretches. It's important when stretching the histogram that you don't "cut into" the histogram itself. That is, avoid moving the left or right sliders inward too far or you'll lose data. In this image, the total histogram stretch is only about 5.5 points on white and black ... not aggressive at all, yet combined with the S-curve to gamma it really improves contrast.
With wavelets, the top slider attacks larger image details, and the successively lower wavelets move down in detail size. I find that minimizing changes to midrange details avoids over-sharpening the details. If your image has noise in it after wavelet sharpening, but is otherwise pleasing, then unclick the top wavelet slider setting and reprocess. This "traps" the noise to a large degree.
And this is how I generally do it with Saturn:
... those alignment boxes are like 30 or 45 pixels square, something like that.
As for VDub, you can do a lot with that. You can even write your own filters. I've found it better, however, to use some of the "canned" filters in iris. It's a command-line-driven scripting language that includes many "filters" of its own. You can use it with DSO images, too.
But that's another story. For now let's stick to your Registax questions.