Best practices for system backups?
I'm taking nightly incremental backups of /home for example to my home storage server and, I guess, due to the databases Firefox uses for various storage, those can't be backed up by just copying them, if Firefox is running and has the database open.
What are the best practices, then, for ensuring all profile data, including these databases, is in a state where it can be backed up, while Firefox is running, in a way it can be restored to a fully functioning profile?
For example, with MySQL databases, you can dump the log before running the backup each night, which basically exports the SQL statements needed so you can skip backing up the actual binary database file (like ibdata1). Does SQLite need something similar? and is this the only special consideration that is needed, or are there other specialized types of data within our profile directories that can't be backed up by simply copying the data while Firefox is running?
Thanks
All Replies (5)
I've always just copied mine and the copies have worked fine when copied back.
Yes, well, that has typically worked in the past, but not so much anymore with these fancy databases. I didn't even need a backup, I just copied my profile folder over to the new computer. Thing is, places.sqlite was secretly corrupted at some point but I was never alerted to this. Everything has appeared and worked fine, my history is all viewable in the old Firefox, nightly backups have been successful, and so on.
Yet, only 12 history items found? Then upon checking integrity in about:profiles the truth was revealed, it was a complete mess. Also tried Firefox Sync, same thing only 12 entries synced. I finally had success with Skyweb's History Export add-on, it was able to grab over 138,000 records and export to json which i was able to import into this new computer.
For databases, in general, it's a bad practice to just plain file copy them while they're still open. Yes, if it's got little to no usage when the copy is performed, you can get lucky and not have issues with restoration. But with these one like places.sqlite that stores all kinds of things besides history, there's no telling when a transaction might be written. The databases have to be backed up in special ways.
Now I'm reading more about different ways that can be executed before my nightly backup runs: .clone, .dump, .backup. An app called Litestream looks to be the best, it looks like it can dump incremental backups of sqlite database which can then be included in nightly system/home incremental backups... will keep reading.
Okay and Thunderbird has a `calendar-data/cache.sqlite` that is 1.6 GB here that is getting backed up 'every night' just for a few KB of changes! That's completely unnecessary people! First, wtf is this cache and why is it so huge? Second, what is the proper way to get incremental backups of all these mozilla databases! I've been swamped with work and haven't figured much out, but surely someone familiar with all their structures should know.
In general, like I do with our MySQL databases at work, those have "binary logging" enabled, which lets me run a command before each nightly system backup that dumps the log file since the last time it was dumped (usually the previous night). Then, this binary log, which only contains the changes sine the previous log was generated and thus orders of magnitude smaller in size, is what gets backed up, while the huge database file itself is excluded from the backups. For the monthly full backups, I think just a "mysqldump" command is run to export the full database in text SQL format, which is backed up instead of the machine code blob (ibdata1 for example), which is actually never backed up. If the backup is needed, MySQL just imports the dump data + the binary logs and you're back up and running. I'm sure there's an equivalent practice we can employ for all these sqlite databases...
If Firefox I would use this Firefox backup helps.
"Back up and restore information in Firefox profiles" https://support.mozilla.org/en-US/kb/back-and-restore-information-firefox-profiles
https://support.mozilla.org/en-US/kb/restore-bookmarks-from-backup-or-move-them
https://support.mozilla.org/en-US/kb/export-login-data-firefox
https://support.mozilla.org/en-US/kb/profiles-where-firefox-stores-user-data
"How to backup History, Bookmarks, Profile, Everything in Firefox on Cloud for emergency restore?" https://support.mozilla.org/bm/questions/966133
There's no on one solution to backup but multiple ways depending on what data you want to restore to start where the sessions was left at.
Well I want everything, like in your first link, and like I'm already doing with my nightly bacula backups. The problem is Firefox has to be closed to be sure that has been done correctly, and that's just not feasible over here to close ff ever. Bacula has some scripts to assist in backing up live MySQL db's correctly, so maybe the community over there knows how to do something similar with SQLite db's...