hello, can you try to replicate this behaviour when you launch firefox in safe mode once? if not, maybe an addon is interfering here...

Troubleshoot extensions, themes and hardware acceleration issues to solve common Firefox problems

I disabled *all* of my add-ons and restarted Firefox, but Options still failed to work at all. Rolled back to FF 37, and all is well again. FF 38 not good.

in a similar thread a user reported that it was a custom theme causing this issue - not sure if that would apply to you as well: https://support.mozilla.org/en-US/questions/1061789

Bingo! The problem was the theme, not an add-on. After much research and experimentation, I found that using about:config to set the value of "browser.preferences.InContent" to false displays Options in the old floating window and allows them to work in my old theme. False was apparently the default before Firefox v38, but for reasons unknown, FF 38 changes the default value to true and thus opens Options in a fixed tab that does not presently work in my preferred theme.

The new default value of true (tab) in FF 38 does work with Firefox's default theme and several other themes, but not my particular theme. For those of us willing to accept Options in the old floating window (false), some old themes can be made to work in Firefox 38. Alas, users who want Options in a tab (true) must either switch to a new theme to stay in FF 38 or else roll back to FF 37 in order to keep their old theme.

I would strongly advise against the option of using an old outdated and insecure Firefox 37.

For now at least flipping the pref and using the old style of options display probably will not cause problems. If problem themes are still supported hopefully they will get fixes.