Firefox shows the "(Never) Remember history" choice if all the history and cookie setting are the default.
Make at least one change to the default setting if you want "Use custom settings for history" to stay selected.
Otherwise Firefox will revert to "(Never) Remember history" if you return to the Privacy tab.
The choice "Use custom settings for history" will stay selected as long as not all settings are the default to make you aware that you made changes.

If you select "Never Remember history" then you enter Private Browsing mode. Be aware that not everything may work as expected in PB mode. All cookies are session cookies that expire if you close Firefox or end that PB mode session. Some other features like disk cache and history are disabled (no visited link change) and some menu entries are grayed (disabled).

Thank you cor-el for your reply. To clarify for any other followers of this post: I found that if I first selected "Remember history" (apparently synonymous with "Default settings") and "OK," then reopened the Options pane and selected "Never remember history," it would stay selected. This seems like a bug to me however, since there is already a selection for single-session Private Browsing in the "Tools" pull-down menu, and it is simply not intuitive that a setting once selected would revert to any other setting simply because you re-opened "Options." This will result in many users who are not paying close attention using settings which they do not intend to use regarding their privacy. This is the sort of thing I expect from Microsoft or Adobe, but not from Mozilla.