Skip to content
Rowsafe
Docs

Protect OpenSearch

Snapshots, Marks, Proof, Rewind, Pulse, and users and roles for OpenSearch 2 and 3.

Rowsafe backs up OpenSearch with its own snapshots, sent to your bucket and encrypted on your server. You get Marks, a weekly Proof, Rewind (bring documents back or rewind the whole server) and Pulse with fixes. Some features aren't there yet: see Not yet.

Restores go back to a snapshot

OpenSearch keeps no change log, so Rowsafe can't restore to any second. You pick a snapshot or a Mark instead:

  • A snapshot is taken every 5 minutes, copying only what's new. You lose at most 5 minutes. Owners can pick 15, 30 or 60 minutes in Settings.
  • A Mark takes one on the spot. Make one before a migration, reindex or bulk delete.

If the snapshot you picked was removed, a copy uses the nearest older one; a rewind in place stops and names it.

Before you start

  • OpenSearch 2 or 3, as one node. Clusters, and servers that answer like Elasticsearch, are refused.
  • Debian or Ubuntu, with the security plugin on or off.
  • REST on 127.0.0.1:9200 by default, over HTTPS (or HTTP if TLS is off).
  • The opensearch program on the server, for Proof and Rewind copies. Backups work without it.
  • A bucket and a passphrase. See Adopt an existing database.

Turn on backups

Run this on the server, then approve it in your browser:

curl -fsSL https://rowsafe.sh | sudo sh -s -- --account

A folder for snapshots. The installer adds /var/lib/rowsafe-opensearch/snapshots to path.repo in opensearch.yml, keeping the old file as opensearch.yml.rowsafe-backup. The agent joins the opensearch group to read snapshots, never to write them.

Rowsafe's own OpenSearch user. With the security plugin on, an administrator (all_access, like admin) signs in once. Rowsafe creates rowsafe with a random password kept on the server. The admin password is never saved.

One restart. OpenSearch reads path.repo only at start. The installer asks first (about a minute; apps reconnect). Say no and backups start after your next restart.

The plan. Review it and say yes. Rowsafe checks the bucket and passphrase, then takes the first snapshot.

  • Without a terminal, use --protect NAME with ROWSAFE_OPENSEARCH_ADMIN_USER and ROWSAFE_OPENSEARCH_ADMIN_PASSWORD (used once). It never restarts OpenSearch.
  • If the demo setup (demo users, demo certificates) is still there, the installer warns and goes on. Pulse shows it until it's gone.

What Rowsafe does

How
BackupsOpenSearch's snapshot of every index and data stream, every 5 minutes, and a full one daily at 01:00 UTC. Only new files are copied. If snapshots get slow, Rowsafe spaces them out to use at most about a quarter of the server's time.
EncryptionThe agent copies new snapshot files to your bucket (or Rowsafe Storage), encrypted on your server. OpenSearch never sees the bucket.
KeepingEvery snapshot for a day, one an hour for a week, and the daily full ones (7 by default) with every Mark since the oldest. Unneeded files are deleted, in the bucket too.
MarksA snapshot taken on the spot.
ProofWeekly: restore the newest snapshot from your bucket into a temporary local OpenSearch, check every index is green with all its documents, then delete it.
RewindA copy, compare, bring documents back, or rewind the whole server with Undo for 7 days (below).
PulseHealth, shards, disk, memory, slow-downs and security (below).
RestartRestart, after you confirm (with root's permission).
Databases & usersIndices and the security plugin's users (below).

Rewind

In the dashboard, for owners and admins.

  • Restore a copy at a snapshot or Mark, as a temporary OpenSearch on the same server that only Rowsafe signs in to.
  • Compare an index with production by _id: deleted, changed and added documents. Large indexes: first million documents (the result says so).
  • Bring back documents into the indices you pick, with their _id. Existing documents are never overwritten unless you include changed ones. A deleted index is made again with the copy's mappings.

Rewind the whole server

Rewind the whole database puts every index and data stream back to a snapshot or Mark:

  1. Rowsafe snapshots everything as it is now, kept 7 days for Undo rewind.
  2. It deletes and restores each index and data stream. Newer indices are removed (they're in the Undo snapshot).

Nothing restarts. Searches of an index fail for the moments it's restored, and writes made meanwhile are replaced. If it fails, the first snapshot is put back.

Pulse

Pulse reads OpenSearch about once a minute: health (green, yellow, red), shards, indices, disk against the watermarks, heap, long garbage collections, refused requests (circuit breakers, full queues), and the last snapshot.

Each fix confirms first and checks the problem is still there:

FindingFix
Health is yellow: replicas have nowhere to go (one server can never hold a copy of its own data)Set replicas to 0 on those indices. Instant; no data moves.
Writes are still refused after the disk filled upAllow writes again: removes the read-only block, only once the disk has room.

The security check covers: security plugin off (also checked from the internet), TLS off, demo users with published passwords, demo certificates, the demo admin certificate (CN=kirk) trusted or its key readable by others, and port 9300 open beyond the server. Fixes need opensearch.yml and a restart, so each shows steps under Do it yourself.

Users and roles

Databases & users uses the security plugin's users. A "database" is an index or data stream.

  • Create an index (empty), optionally with an owner. Remove one: Rowsafe saves a Mark first.
  • Add a user on the indices or patterns you list (logs-*): read only, read and write (also documents, creating those indices, mappings) or owner (everything on them).
  • New password and Remove user. Passwords are made on your server and shown only to you.

Built-in users (admin, kibanaserver...), reserved and hidden users, administrators (all_access) and Rowsafe's own are listed but never changed. Without the security plugin there are no users.

What Rowsafe may do

Rowsafe's user (rowsafe) reads health and stats, and manages snapshots in its own folder. Only when someone asks, it creates, writes and deletes indices, changes replicas, removes the read-only block, and manages users (its role is in plugins.security.restapi.roles_enabled).

Root decides what Rowsafe may do on the server (permissions, sudo rowsafe-allow): restart allows Restart. Rowsafe doesn't install OpenSearch updates yet.

On servers Rowsafe creates

Rowsafe Cloud and Create a server for me can install OpenSearch 3:

  • OpenSearch's own packages (kept on 3.x), one node, with its bundled Java.
  • Security plugin on, no demo users or certificates. One admin, password readable by root only.
  • HTTPS on 9200. Port 9300 stays on the server.
  • Half the server's memory (at most 31 GB), Performance Analyzer off. 4 GB of memory or more.
  • No standby, clones or private connections from AWS.

Apps connect over HTTPS on 9200 with a user from Databases & users.

Restore without Rowsafe

With the ROWSAFE_REPO_* lines of /etc/rowsafe/agent.env in the environment, decrypt a snapshot into a folder:

set -a; . /etc/rowsafe/agent.env; set +a
# --stanza is the database's "Bucket folder" (on its Settings page in the dashboard).
rowsafe-agent opensearch download-backup --stanza search
rowsafe-agent opensearch download-backup --stanza search --label 20261009-013000F --to /srv/os-restore

It's a normal snapshot repository. Make it readable by OpenSearch, add it to path.repo and restart. The agent prints the two requests that register and restore it.

Not yet

  • Restores to any second, and Find the moment.
  • Updates and upgrades from the dashboard, Tuning, logs and recommendations.
  • Standby servers, clones, Move in, safe copies and migration previews.
  • Connection pooling, a second copy, and Files.

Limits

  • Restores go back to a snapshot (every 5 minutes by default) or a Mark, never in between.
  • One node only. No clusters, no Elasticsearch.
  • Not in Docker yet. No OpenSearch agent image: install the agent on the server.
  • Disk: snapshots are kept on the server too (/var/lib/rowsafe-opensearch/snapshots).
  • Users aren't in backups. Security plugin users and roles and other system indices stay out. Rewinds leave index templates and ingest pipelines as they are now.
  • Temporary servers (Proof, copies) need about 700 MB of free memory. The agent refuses if there isn't enough.
  • Compare looks at the first million documents of an index.
  • Data streams only take new documents, so changed ones are never overwritten.
  • Rewinding the whole server makes searches of each index fail briefly and replaces writes made meanwhile.
  • Security fixes need opensearch.yml and a restart: steps you do yourself.
  • The first restart: backups start only after OpenSearch restarts once with Rowsafe's folder in path.repo.
Edit on GitHub