Snapshots in Elastic Search 7

Cesar Capillas

How to backup data in Elastic

Snapshots are a very important task for backup, restoration and upgrade procedures in Elastic Search. A snapshot is just a backup taken from a running ES cluster, what we often know as a hot backup. A big disclaimer is taken from documentation; you cannot back up an Elasticsearch cluster by simply taking a copy of the data directories of all of its nodes, while it is running. The only reliable way to back up a cluster is by using the snapshot and restore functionality.

Snapshots are taken incrementally for a set of defined indices via Snapshot API or Kibana snapshot interface, where it is also possible to automate the snapshots. It is necessary to register a repository configuration path for saving the corresponding snapshots. The rootpath for snapshots’ repositories may be defined locally, for example with a dedicated backup disk or mount point in Elastic server. This config is set in your elasticsearch.yml config file (or in your $JAVA_OPTS via -D option) with:

path.repo: /opt/data/snapshots

There are also plugins that support remote repositories on S3, HDFS, Azure and Google Cloud Storage.

In the Kibana interface, it is quite simple to register a shared filesystem repository, where we can set some options as chunk size or compression of the snapshots.

If you prefer the Kibana Dev Console or a REST Client (even curl if you are a command line hero), you may do something like:

# Creating a repo called test under path.repo
PUT /_snapshot/test
{
  "type": "fs",
  "settings": {
    "location": "/opt/data/snapshots/test"
  }
}

GET /_snapshot/test

# Creating a snapshot called snapshot.20191213
PUT /_snapshot/test/snapshot.20191213?wait_for_completion=true
{
  "indices": "filebeat-alfaudit-*,filebeat-alfresco-*",
  "ignore_unavailable": true,
  "include_global_state": false
}

# Get status
GET /_snapshot/test/snapshot.20191213/_status

# Delete your snapshot test
DELETE /_snapshot/test/snapshot.20191213

Finally, you can automate the generation of snapshots in Kibana UI:

Snapshots status may be checked via API or Elastic logs:

[2019-12-15T02:30:00,003][INFO ][o.e.x.s.SnapshotLifecycleTask] [pudu1] snapshot lifecycle policy [daily-snapshot] issuing create snapshot [elk741-2019.12.15-gjudab7hqgod6caapoazew]
[2019-12-15T02:30:00,005][INFO ][o.e.x.s.SnapshotLifecycleTask] [pudu1] snapshot lifecycle policy job [daily-snapshot-1] issued new snapshot creation for [elk741-2019.12.15-gjudab7hqgod6caapoazew] successfully
[2019-12-15T02:30:07,642][INFO ][o.e.s.SnapshotsService   ] [pudu1] snapshot [elk741:elk741-2019.12.15-gjudab7hqgod6caapoazew/q30iu1zITl68Zwm6D-PSYg] started
[2019-12-15T02:32:36,673][INFO ][o.e.s.SnapshotsService   ] [pudu1] snapshot [elk741:elk741-2019.12.15-gjudab7hqgod6caapoazew/q30iu1zITl68Zwm6D-PSYg] completed with state [SUCCESS]

Links: