A lot depends on the camera you're going to be using, but in the main you want as much focal length as your mount will support and your aperture will support.
The best lunar and planetary images being made today by amateurs are using Celestron C-14-class scopes and imagers like the Lumenera 2-1 and webcams or DMK/DBK firewirecams. There are also some purpose-built systems like Paolo Lazarottis Cassegrain "cameras" and large-aperture dobsonians equipped with tracking platforms and webcams or DSLRs (a'la Wesley Higgins).
I have used a very fine 1,200mm-focal-length 8" apo for my best images to date. That's an F8 native system. However, most of those were made using at least a 2.5X Powermate to extend the focal length, and some of the best one were made between 4,000mm and 6,000mm focal length (extended using barlows). So, the "best" results with that particular system are achieved at F20 to F40.
If you use a DSLR, you need even more focal length to get a good image scale at the same focal length. Put another way, a DSLR will give better results on the same scope at less than optimal seeing (due to the lesser magnification).
If you use a DSLR or other large-chip camera, you need at least a 2" throughput for your imaging train. So, if you're starting from scratch with a setup to do lunar and planetary imaging, I'd suggest something like an 8" SCT with a 2" visual back as a starting point using off-the-shelf, readily-available, affordable equipment.
To get an idea of what's possible, look in the Online Reader Gallery here to see what others are doing with specific equipment setups. For a more in-depth treatment of this subject, see Robert Reeves' excellent guide (click here).