I get a tiny new window for Pop out; it seems to match the resolution of the video.

Could you check some settings in Firefox's about:config preferences editor?

(1) In a new tab, type or paste about:config in the address bar and press Enter. Click the button promising to be careful.

(2) In the filter box, type or paste link.open and pause while the list is filtered

You may want to reset any settings that are not set to the default (the non-default items should appear in bold). I think the second one is the critical one in this case.

browser.link.open_newwindow

3 = divert new window to a new tab (default) (checked*)
2 = allow link to open a new window (unchecked*)
1 = force new window into same tab

* First checkbox in Options > Tabs (on Mac, Preferences > Tabs)

browser.link.open_newwindow.restriction

0 = open_newwindow preference honored for all new windows
2 = open_newwindow preference honored for all new windows except script windows with features (default)
1 = open_newwindow preference ignored completely (same effect as open_newwindow=2)

browser.link.open_newwindow.override.external

-1 = open_newwindow applies to external links (default)
3 = open external links in a new tab
2 = open external links in a new window
1 = open external links in active tab

If you have any add-ons that alter the way tabs work in Firefox, they might be overriding or changing these settings, so check those as well.

Thank you so much! I had "browser.link.open_newwindow.restriction" set to 0. Changing it to 2 fixed the YouTube pop out windows.

Cheers!