The setting in Firefox/Tools > Options > Tabs: "New pages should be opened in": "a new tab/window" is for links that specify a target window to open the link. That option allows to divert such links to a new tab instead of a new window. Links that do not specify a where will open in the active tab unless told to do something else by JavaScript.

It is possible that an extension is interfering.

For links that do not specify a where, hold down Ctrl and left-click the link or use the right-click context menu to open the link in a new tab or window. For Mac use the command key in place of Ctrl.

Try starting Firefox in safe mode. Hold shift when opening the desktop icon. -or- click Restart with Add-ons disabled under the Help menu.

You can override how links are opened via the browser.link.open_newwindow pref.

• 1: current tab; 2:new window; 3:new tab;

• http://kb.mozillazine.org/browser.link.open_newwindow (3)

See this pref for the behavior of JavaScript links:

• http://kb.mozillazine.org/browser.link.open_newwindow.restriction (0)

• http://kb.mozillazine.org/about:config