Apple Users

I don't actually use Handbrake to do the ripping, just the encoding. I use a cross-platform program named MakeMKV to rip DVDs and blu-rays, then feed those ripped files to Handbrake.

So we'll break this down into two steps: 1) rip; 2) encode. We'll be able to check at each step to see where the problem is.

You'll need to download & install MakeMKV from makemkv.com
MakeMKV is "free while in beta", and it's been in beta for years. I myself have purchased a license because I use it often. For those wanting to test it, the MakeMKV devs publish an unlock code here that unlocks the software for a month or so. When the current code expires they post a new one, although you may find that you have a few days where the old key expired and they haven't posted the new one yet.

Once you've installed MakeMKV, choose "Register" from the Help menu and paste the unlock code in. You'll be asked to quit & restart the application.

With MakeMKV running, load your DVD. There should be a graphic of an optical drive with a DVD in it, click the image and MakeMKV should scan the disc and present you with a list of titles on the disc. Select the titles you want to copy from the disc to your hard disk, then click the "save selected titles" button (looks like a green arrow pointing at a hard disk). You'll be asked to select a save location.

Once that has finished, play the file with VLC. If everything is OK, try encoding that file with Handbrake.

Ripping & video encoding are both complicated topics, so if you would like more help I'm happy to do so. Since we're both using macOS, we could even do a screenshare+audio where I can see what's on your screen and talk to you in real time.

Good luck & let me know how it goes!