How to backup data in Elastic
Snapshots are a very important task for backup, restoration and upgrade procedures in Elastic Search. A snapshot is just a backup taken from a running ES cluster, what we often know as a hot backup. A big disclaimer is taken from documentation; you cannot back up an Elasticsearch cluster by simply taking a copy of the data directories of all of its nodes, while it is running. The only reliable way to back up a cluster is by using the snapshot and restore functionality.
Snapshots are taken incrementally for a set of defined indices via Snapshot API or Kibana snapshot interface, where it is also possible to automate the snapshots. It is necessary to register a repository configuration path for saving the corresponding snapshots. The rootpath for snapshots’ repositories may be defined locally, for example with a dedicated backup disk or mount point in Elastic server. This config is set in your elasticsearch.yml config file (or in your $JAVA_OPTS via -D option) with:
path.repo: /opt/data/snapshots
There are also plugins that support remote repositories on S3, HDFS, Azure and Google Cloud Storage.
In the Kibana interface, it is quite simple to register a shared filesystem repository, where we can set some options as chunk size or compression of the snapshots.
If you prefer the Kibana Dev Console or a REST Client (even curl if you are a command line hero), you may do something like:
# Creating a repo called test under path.repo PUT /_snapshot/test { "type": "fs", "settings": { "location": "/opt/data/snapshots/test" } } GET /_snapshot/test # Creating a snapshot called snapshot.20191213 PUT /_snapshot/test/snapshot.20191213?wait_for_completion=true { "indices": "filebeat-alfaudit-*,filebeat-alfresco-*", "ignore_unavailable": true, "include_global_state": false } # Get status GET /_snapshot/test/snapshot.20191213/_status # Delete your snapshot test DELETE /_snapshot/test/snapshot.20191213
Finally, you can automate the generation of snapshots in Kibana UI:
Snapshots status may be checked via API or Elastic logs:
[2019-12-15T02:30:00,003][INFO ][o.e.x.s.SnapshotLifecycleTask] [pudu1] snapshot lifecycle policy [daily-snapshot] issuing create snapshot [elk741-2019.12.15-gjudab7hqgod6caapoazew] [2019-12-15T02:30:00,005][INFO ][o.e.x.s.SnapshotLifecycleTask] [pudu1] snapshot lifecycle policy job [daily-snapshot-1] issued new snapshot creation for [elk741-2019.12.15-gjudab7hqgod6caapoazew] successfully [2019-12-15T02:30:07,642][INFO ][o.e.s.SnapshotsService ] [pudu1] snapshot [elk741:elk741-2019.12.15-gjudab7hqgod6caapoazew/q30iu1zITl68Zwm6D-PSYg] started [2019-12-15T02:32:36,673][INFO ][o.e.s.SnapshotsService ] [pudu1] snapshot [elk741:elk741-2019.12.15-gjudab7hqgod6caapoazew/q30iu1zITl68Zwm6D-PSYg] completed with state [SUCCESS]
Links: