Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Backups: Schedule automatic index database and album backups #4243

Closed
lastzero opened this issue May 10, 2024 · 3 comments
Closed

Backups: Schedule automatic index database and album backups #4243

lastzero opened this issue May 10, 2024 · 3 comments
Assignees
@lastzero @graciousgrey
Labels
config Global Configuration, Initialization, and User Settings enhancement Refactoring, improvement or maintenance task feedback wanted Please let us know your thoughts released Available in the stable release

Comments

@lastzero
Copy link
Member

lastzero commented May 10, 2024 •
edited

As a user who doesn't create backups in any other way, I want PhotoPrism to automatically create backups for me at configurable intervals so that I can restore the index if my database gets lost or corrupted.

While a backup worker has already been implemented for this, we still need to decide on the configuration options to automatically run it in the background, as users likely have different preferences. Feel free to add a comment below to let us know what options you would need for your own use.

Related Issues:
@lastzero lastzero added the enhancement Refactoring, improvement or maintenance task label May 10, 2024
@lastzero lastzero self-assigned this May 10, 2024
lastzero added a commit that referenced this issue May 10, 2024
@lastzero
Backups: Add worker to create backups in the background #4237 #4243
424ed84 
In the next step, this worker can be configured to automatically create
index and/or album backups at regular intervals.

Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero lastzero changed the title Backups: Add a background worker that creates backups at configurable intervals Backups: Automatically create index and/or album backups in the background May 10, 2024
@lastzero lastzero added the in-progress Somebody is working on this label May 10, 2024
@lastzero lastzero changed the title Backups: Automatically create index and/or album backups in the background Backups: Automatically create index and/or album backups May 10, 2024
@lastzero lastzero added waiting Impediment / blocked / waiting help wanted Well suited for external contributors! feedback wanted Please let us know your thoughts config Global Configuration, Initialization, and User Settings and removed in-progress Somebody is working on this waiting Impediment / blocked / waiting labels May 10, 2024
lastzero added a commit that referenced this issue May 11, 2024
@lastzero
Config: Allow scheduling of indexing and backup tasks #2495 #2608 #4243
0e7c91f 
Note that this is "bleeding edge" functionality and that the newly added
config option PHOTOPRISM_BACKUP_RETAIN can be set, but does not have any
effect yet. Feedback welcome!

Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero
Copy link
Member Author

lastzero commented May 11, 2024 •
edited

Here is an overview of the updated options for configuring storage, backups, and index workers:

Storage

Environment CLI Flag Default Description
PHOTOPRISM_CONFIG_PATH --config-path config storage PATH, values in options.yml override CLI flags and environment variables if present
PHOTOPRISM_DEFAULTS_YAML --defaults-yaml /etc/photoprism/defaults.yml load config defaults from FILE if exists, does not override CLI flags and environment variables
PHOTOPRISM_ORIGINALS_PATH --originals-path storage PATH of your original media files (photos and videos)
PHOTOPRISM_ORIGINALS_LIMIT --originals-limit 1000 maximum size of media files in MB (1-100000; -1 to disable)
PHOTOPRISM_RESOLUTION_LIMIT --resolution-limit 150 maximum resolution of media files in MEGAPIXELS (1-900; -1 to disable)
PHOTOPRISM_USERS_PATH --users-path users relative PATH to create base and upload subdirectories for users
PHOTOPRISM_STORAGE_PATH --storage-path writable storage PATH for sidecar, cache, and database files
PHOTOPRISM_SIDECAR_PATH --sidecar-path custom relative or absolute sidecar PATH optional
PHOTOPRISM_SIDECAR_YAML --sidecar-yaml true create YAML sidecar files to back up picture metadata
PHOTOPRISM_CACHE_PATH --cache-path custom cache PATH for sessions and thumbnail files optional
PHOTOPRISM_IMPORT_PATH --import-path base PATH from which files can be imported to originals optional
PHOTOPRISM_IMPORT_DEST --import-dest relative originals PATH to which the files should be imported by default optional
PHOTOPRISM_ASSETS_PATH --assets-path assets PATH containing static resources like icons, models, and translations
PHOTOPRISM_TEMP_PATH --temp-path temporary file PATH optional

Backup

Environment CLI Flag Default Description
PHOTOPRISM_BACKUP_PATH --backup-path custom base PATH for creating and restoring backups optional
PHOTOPRISM_BACKUP_SCHEDULE --backup-schedule daily backup SCHEDULE in cron format (e.g. "0 12 * * *" for daily at noon) or at a random time (daily, weekly)
PHOTOPRISM_BACKUP_RETAIN --backup-retain 3 NUMBER of index backups to keep (-1 to keep all)
PHOTOPRISM_BACKUP_DATABASE --backup-database true create regular backups based on the configured schedule
PHOTOPRISM_BACKUP_ALBUMS --backup-albums true create YAML files to back up album metadata

Indexing

Environment CLI Flag Default Description
PHOTOPRISM_INDEX_WORKERS, PHOTOPRISM_WORKERS --index-workers 4 maximum NUMBER of indexing workers, default depends on the number of physical cores
PHOTOPRISM_INDEX_SCHEDULE --index-schedule indexing SCHEDULE in cron format (e.g. "@every 3h" for every 3 hours; "" to disable)
PHOTOPRISM_WAKEUP_INTERVAL --wakeup-interval 15m0s TIME between facial recognition, file sync, and metadata worker runs (1-86400s)
PHOTOPRISM_AUTO_INDEX --auto-index 120 delay before automatically indexing files in SECONDS when uploading via WebDAV (-1 to disable)

@lastzero lastzero added in-progress Somebody is working on this please-test Ready for acceptance test and removed help wanted Well suited for external contributors! labels May 11, 2024
@lastzero lastzero mentioned this issue May 12, 2024
Closed
lastzero added a commit that referenced this issue May 12, 2024
@lastzero
Backups: Add config option to limit the number of backups to keep #4243
1fe0bab 
PHOTOPRISM_BACKUP_RETAIN lets to specify the number of index database
dumps to keep (backup filenames are in the format "YYYY-MM-DD.sql").

Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero
Copy link
Member Author

lastzero commented May 12, 2024 •
edited

Support for the newly added environment variable PHOTOPRISM_BACKUP_RETAIN is now also ready for testing. It allows you to specify the number of index database backup files to keep. Since the file names of the automatically created SQL dumps have the format YYYY-MM-DD.sql, old backups can be easily found and removed.

@lastzero lastzero removed the in-progress Somebody is working on this label May 12, 2024
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
Backups: Refactor config and add "sidecar-yaml" config option #4243
0396e86 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
Backups: Improve usage descriptions of config options #4243
b3782fc 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
Backups: Update frontend translations.pot #4243
6846e62 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
Backups: Update frontend translations.json and package-lock.json #4243
598624e 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
Backups: Update frontend translations.json after merge #4243
5c7bb3b 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
Config: Change defaults for BackupIndex and BackupRetain #4243
7805280 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
Config: Improve options report and change BackupRetain default #4243
db5336a 
Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero
Copy link
Member Author

We changed the PHOTOPRISM_BACKUP_RETAIN default value from 14 to just 3 (backups to keep), after finding out how much disk space the dumps might occupy. My tests with these new defaults went smoothly so far, although I haven't properly tested it with SQLite yet (backups are created at noon by default to prevent NAS devices from waking up during the night):

backups

If you do not want automatic backups to be created for your instance, you can deactivate them under Settings > Advanced:

database-backups-advanced

I'll leave further testing to @graciousgrey and our community so I can move on to other issues. Thank you very much!

@graciousgrey graciousgrey self-assigned this May 13, 2024
lastzero referenced this issue May 13, 2024
@lastzero
Backups: Improve log messages for photo and album metadata exports #3284
f00d88d 


Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
Index: Improve log message when writing metadata to sidecar file #4243
b0f42ed 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 13, 2024
@lastzero
API: Improve log message when writing metadata to sidecar file #4243
52b181b 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 14, 2024
@lastzero
Backups: Improved saving of photo and album YAML files #4243
9527082 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 14, 2024
@lastzero
Backups: Rename "backup-index" config option to "backup-database" #4243
d5580c1 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 14, 2024
@lastzero
Backups: Skip unchanged albums when saving YAML backup files #4243
c73a2e3 
Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero lastzero changed the title Backups: Automatically create index and/or album backups Backups: Schedule automatic index database and album backups May 14, 2024
lastzero added a commit that referenced this issue May 14, 2024
@lastzero
Backups: Update YAML file backups when albums are deleted #4243
4e7a3c7 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 14, 2024
@lastzero
Config: Update "sidecar-yaml" usage description #4243
bb98e7a 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 14, 2024
@lastzero
Moments: Reduce activity of background workers #4237 #4243
7bec344 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 15, 2024
@lastzero
Backups: Improve handling of custom backup file and path names #4243
6a1c92a 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 15, 2024
@lastzero
Backups: Improve command-line backup and restore commands #4243
ca78305 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 19, 2024
@lastzero
Backups: Change file permissions for index database dumps #4243
c2ef35d 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 19, 2024
@lastzero
Config: Add “daily” and “weekly” backup schedule options #4243
3d908c7 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 19, 2024
@lastzero
Config: Improve usage descriptions for backup options and flags #4243
6f00545 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 19, 2024
@lastzero
Config: Update docker-compose.yml example files #4243
66a739d 
Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue May 19, 2024
@lastzero
Config: Update docker-compose.yml example files #4243
2f246c2 
Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero lastzero mentioned this issue May 20, 2024
Closed
lastzero added a commit that referenced this issue May 20, 2024
@lastzero
Config: Add backup options to the Advanced Settings UI #2495 #4243 #4269
2caf04a 


Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero lastzero mentioned this issue May 20, 2024
Open
@graciousgrey graciousgrey added tested Changes have been tested successfully and removed please-test Ready for acceptance test labels May 22, 2024
@lastzero lastzero mentioned this issue May 24, 2024
Open
@lastzero lastzero added released Available in the stable release and removed tested Changes have been tested successfully labels May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lastzero lastzero

@graciousgrey graciousgrey

Labels
config Global Configuration, Initialization, and User Settings enhancement Refactoring, improvement or maintenance task feedback wanted Please let us know your thoughts released Available in the stable release
Projects
Roadmap 🚀✨
Status: Release 🌈
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lastzero @graciousgrey