I read up on it in there wiki and they said to fix the frame rate isntead of use same as source, Its supposed to only effect mixed frame rate content but did that and cranked up the B Pyramid number, Motion Estimation numbers, and a few other things and it looks like it resolved it.
It depends on the source. If your source is a videogame, then use the same framerate as the source. I.E. If you capture at 30 FPS, make sure you keep it at 30 FPS. If you do the motion blur technique where you capture at 150 or 300 FPS, then you'll use Avisynth to take every 5th or 10th frame (150 / 5 = 30 and 300 / 10 = 30).
The mixed content they refer to has to do with DVDs. NTSC movie DVDs are 29.970 FPS, but drop down to 23.976 FPS for progressive content. Interlaced content stays higher. PAL is simply 25 FPS. You shouldn't mess with the framerate unless you are ripping DVD movies. Getting into that, though, will usually require additional steps. You simply can't convert 23.976 FPS to 25 FPS (or vice versa) and have it be smooth.
HandBrake is not neary as good at giving a max quality file at a low size if you put in a larger than needed target file size. It gave me a file 2x larger than my XVID file.
It's all in the settings. I've never used HandBrake, but according to their website, they link against x264. So the quality is all in the settings still.
So now I am experimenting with average bit rate numbers to find a good balance of quality & speed.
Off the top of your head know a good ABR for 1280x720 footage like these HD game videos to where it would appear to the eye the same as lossless?
I use CRF rather than ABR. ABR will require multiple passes to get the same quality. There are benefits to both, though. ABR will allow you to target a specific filesize. If you want your final filesize to be 200 MB, then you can easily target that with ABR. You will have to make multiple passes for the video to look good. This also requires extra time to encode because of the additional passes.
With CRF, you target a specific quality instead (CRF = Constant Rate Factor). You won't know the final filesize of the target, but you only need to make one pass to get quality that is 99.9% on par with ABR.
Typical settings that are used for bitrate settings varies on the resolution. For great looking 720p content, you'll want to target the 10-12MB range. And for 1080p, you'll want to target 12-15MB. This typically isn't ideal for the internet. YouTube is just going to cut the bitrate down to 600KB or so. If you stream the file yourself or host it on some higher-quality streaming site, then it will be more worth it.
When making my videos, I usually set CRF between 22 and 25. The lower the number, the better the quality. Naturally, encoding time will increase, as well as the filesize. There's a tradeoff for everything.
Edit: I am happy with ABR of 8000 it made a 40mb file, I ran a target file size test for XVID and it looks the same to me! I know your going to kill me and say its not possible but it does.
It all depends on your settings. If you're using maxed out settings with XVID, and terrible settings for H.264, then XVID might definitely look better. You'll really need to play around with settings.
Here's an example of reasonable/mid-quality settings with x264:
--bframes 3 --b-pyramid --ref 4 --crf 25 --weightb --subme 7 --8x8dct
Here's an example of unreasonable/insane-quality settings with x264:
--bframes 16 --b-pyramid --direct auto --ref 16 --crf 24 --partitions all --weightb --me tesa --subme 9 --mixed-refs --8x8dct --no-fast-pskip --no-dct-decimate --trellis 2 --qpmin 0