See also http://kb.mozillazine.org/Register_protocol

Thanks for your reply.

That's link outlines exactly what I've tried, but isn't working. Trowling through bugzilla shows others with issues (over 100 bugs for a "protocol handler" search), but most are pretty old.

I'm fine with opening a new bugzilla ticket, but I want to try and get a feel for if I've even in the right area still, or if this behavior was long depreciated. I get the feeling that the about:config settings were depreciated in favor of just fallback to the OS (since that worked previously), but that is seemingly ignored now on Linux.

Dear people from the future,

There didn't seem to be any good information out there on this and I just had to figure it out. Seems like at some point the about:config tweaks documented all over the place stopped working. The only way I've found to get newer firefox versions to register a custom protocol is to specify a desktop file for the application you want to launch.

The secret sauce is to create a .desktop file in /usr/share/applications for your application (see the existing ones as examples, especially the firefox one for http support). Then add a line to defaults.list (same directory) similar to the other x-scheme-handler lines that are already there, just substitute your protocol and desktop file.

The next time you click on one of the links for the custom protocol you should get the appropriate options to use the registered app to handle the URI.