An Oracle Cloud free ARM instance that had been running for years was disabled by Oracle and could not start again.

The boot volume was still healthy, but I tested all three Availability Domains in the Frankfurt region. None had enough ARM capacity at the time, so the boot volume could no longer be started with its original shape.

At that point, restoring the original server was no longer a practical goal. The priority became recovering the data.

Preserve the boot volume

Before terminating the instance, I recorded the OCIDs of its boot and data volumes. I then terminated it while explicitly preserving the boot volume:

oci compute instance terminate `
  --instance-id $instanceId `
  --preserve-boot-volume true `
  --region "eu-frankfurt-1" `
  --force `
  --wait-for-state TERMINATED

The important option is:

--preserve-boot-volume true

Termination is not required if the instance is already stopped and you prefer to detach its boot volume directly. In my case, the old instance was no longer useful, so I terminated it to release the compute resource.

Afterward, I confirmed that the boot volume had reached the AVAILABLE state:

oci bv boot-volume get `
  --boot-volume-id $bootVolumeId `
  --region "eu-frankfurt-1"

Attach it to a rescue VM

An OCI volume can only be attached directly to an instance in the same Availability Domain. I attached the old boot volume to another Ubuntu VM as a read-only data disk:

oci compute volume-attachment attach `
  --instance-id $targetInstanceId `
  --volume-id $bootVolumeId `
  --type paravirtualized `
  --is-read-only true `
  --region "eu-frankfurt-1" `
  --wait-for-state ATTACHED

On the rescue VM, lsblk showed the current system disk as /dev/sda and the old boot volume as /dev/sdb:

sudo lsblk -o NAME,SIZE,FSTYPE,LABEL,UUID,MOUNTPOINT

The old root filesystem was /dev/sdb1, so I mounted it without allowing writes or ext4 journal replay:

sudo mkdir -p /mnt/oldroot
sudo mount -t ext4 -o ro,noload /dev/sdb1 /mnt/oldroot

The complete filesystem from the old server was now available under /mnt/oldroot.

Recover the Docker PostgreSQL volume

The original Docker Compose file used a named volume:

services:
  db:
    volumes:
      - pgdata:/var/lib/postgresql/data

Docker named volumes are directories on the host, normally stored under:

/var/lib/docker/volumes/<volume_name>/_data

I located the PostgreSQL data directory by searching for PG_VERSION:

sudo find /mnt/oldroot -xdev \
  -type f -name PG_VERSION \
  -print 2>/dev/null

That file also identifies the PostgreSQL major version required for recovery.

Because the old server had not shut down cleanly, PostgreSQL needed to perform WAL crash recovery. That process writes to the data directory, so I did not start it against the original read-only disk.

Instead, I copied the directory while preserving ownership and permissions:

sudo mkdir -p /srv/pgrecover

sudo cp -a \
  /mnt/oldroot/var/lib/docker/volumes/<volume_name>/_data \
  /srv/pgrecover/

Do not use chmod -R 777 to work around permission errors. PostgreSQL checks its data directory permissions, and changing them can prevent it from starting.

I then launched a temporary container using the same PostgreSQL major version and image family:

docker run -d \
  --name pgrecover \
  -v /srv/pgrecover/_data:/var/lib/postgresql/data \
  postgres:<major_version>-alpine

The logs showed PostgreSQL replaying WAL and becoming ready:

docker logs -f pgrecover

redo starts at
redo done
database system is ready to accept connections

Verify the data and create a portable dump

I checked the databases, tables, and several important record counts:

docker exec -it pgrecover \
  psql -U <db_user> -d <db_name> -c '\dt'

docker exec -it pgrecover \
  psql -U <db_user> -d <db_name> \
  -c "select count(*) from <important_table>;"

Once the data looked correct, I created a custom-format logical dump:

docker exec pgrecover \
  pg_dump -U <db_user> -d <db_name> \
  -Fc -f /tmp/database.dump

docker cp \
  pgrecover:/tmp/database.dump \
  ./database.dump

Finally, I verified that PostgreSQL could read the dump:

pg_restore -l ./database.dump

A backup file merely existing does not prove that it can be restored. For important data, the best verification is still an actual test restore into an empty database.

Do not forget the state outside PostgreSQL

The database was only part of the recovery.

The server’s environment file contained application credentials and an encryption key. Without that key, some successfully restored database fields would still have been unreadable.

User-uploaded files also lived on the host filesystem while the database stored only their references. Restoring PostgreSQL without those directories would have left the application with valid records pointing to missing files.

One useful place to look is the deployment script’s exclude list. Files and directories deliberately excluded from deployment are often exactly the production-only state that needs separate backup.

Result

The original OCI instance never returned, but its data did:

• the boot volume was preserved;
• it was mounted read-only on another VM;
• the Docker PostgreSQL volume was copied;
• PostgreSQL completed WAL recovery on the copy;
• the data was checked and exported;
• the dump was verified;
• application keys and uploaded files were recovered separately.

picture by Sergio Mena Ferreira