See screenshot of Tab settings.

I have this problem as well and the Tab settings did not work. I have Firefox pinned to my taskbar. I had no problems but my daughter installed a photo editor and all of the sudden when I click the shortcut icon on the taskbar, a new Firefox icon pops up and the new windows open there. Each time I click the icon, it just opens a new window. This never happened before last week, so I know there's a setting thats off somewhere, I just can't seem to find it.

The above problem seemed to right itself for me as it no longer happens. I hope the same happens for you.

I had so many problems with multiple windows and losing windows and tabs when I first started to use Windows 11. I started to check open windows all the time before I shut down, by hovering over the browser in the Taskbar.

I have used all Windows versions since Windows 95 and was usually used to a new version within a few weeks. I hate Windows 11 as it is not an improvement on Windows 10. I'm beginning to think I would be better off using a large iPad so avoid the problems I have had since I bought my current Windows 11 PC nearly 2 years ago. I like the physical keyboard plus the larger screen. that is all that is keeping me with Windows.

Check out Refresh Firefox. https://support.mozilla.org/en-US/kb/refresh-firefox-reset-add-ons-and-settings I have used it in the past when my FF settings were correct but it seemed to ignore them. Good luck.