You can delete the content-prefs.sqlite file in the Profile Folder to only remove settings stored in this file. That way you keep exceptions stored in permissions.sqlite that would get lost if you clear the "Site Preferences" via Clear Recent History.

You can use this button to go to the currently used Firefox profile folder:

• Help > Troubleshooting Information > Profile Directory: Show Folder (Linux: Open Directory; Mac: Show in Finder)
• http://kb.mozillazine.org/Profile_folder_-_Firefox

• https://developer.mozilla.org/Mozilla/Preferences/Preference_reference/browser.download.lastDir.savePerSite

Thanks for the response. So my only option is to delete this file after closing out of firefox? There's no way to simply remove the ability to save the per-site location in the first place?

If you have visited a website before then it is likely that Firefox has stored some folders in the past and may still be using them despite having created the browser.download.lastDir.savePerSite pref. If that is the case then you need to reset all stored settings. You can do that by clearing the Site Preferences, but that will also remove all permissions that are stored in permissions.sqlite. Deleting content-prefs.sqlite will only remove data that is stored in this file.

Clearing "Site Preferences" clears all exceptions for cookies, images, pop-up windows, software installation (permissions.sqlite), and passwords (logins.json), and other website specific data (content-prefs.sqlite).