I don't know how old or computer expert your children are but you might like to start with this recent discussion by the ed-meister and me -

https://support.mozilla.com/en-US/questions/825529

In short, I don't know any way to solve the problem except by supervision or a password on your operating system(not on Firefox).

It is, of course, possible to delete any individual history items, cookies and so on. And heaven knows what you can do with a USB stick.

Another couple of thoughts-

I have read a suggestion that some routers can keep a history record.

At some time you probably should hold a family discussion on this subject. "Trust" will have to come into it. You'll have to trust the children and they'll need to know that you trust them. Good luck with that, Naive, if you like.

Be aware that if it is possible to remove or disable "Private Browsing" then that still won't make it impossible to delete private data like visited websites (history) or cache or cookies.
There are a lot of ways to clear such data, either via Delete browsing, search and download history on Firefox or directly by deleting the History in the Library or sidebar.
Private Browsing makes it easier because all data is kept in memory and nothing is saved to disk, so nothing needs to be removed.

If you want to see which sites have been visited then you should look at others means outside Firefox like a router or firewall with a password protected logging feature that can't be bypassed or a program like Wireshark.
Wireshark: http://www.wireshark.org/download.html

See also http://kb.mozillazine.org/Parental_controls

If you are still about, you might like to look at the JScher2000 reply on this newer thread -

https://support.mozilla.com/en-US/questions/827825

And these from Xircal -

https://support.mozilla.com/en-US/questions/828454

Thanks for all your help. I think I'll go the internet filter route and see how that goes.