Iresh Dudeja
Projects

Moving servers between AWS accounts and regions

I owned the project and ran the move. A colleague wrote the plan.

Each customer server is a single pod on Kubernetes, with its data on one Amazon EBS disk.

Why it was needed

Moving some servers to another existing cluster would let the first one shrink. The two clusters were in different AWS accounts and regions, and a plain restore from backup cannot carry a disk across both.

Region A · account ARegion B · account BshareCustomer serverstopped, backed upEBS snapshotfrom the backupCopy scriptre-encrypt with our keyNew diskin the node's zoneCustomer serverattached, started
The disk moves as a snapshot. It is re-encrypted with our own key so the other account can use it, then becomes a new disk for the server.

How it was done

  1. Moved one test server by hand and wrote the method down as a runbook.

  2. Stopped each server and took a Velero backup (a Kubernetes backup tool) of its disk.

  3. Copied the snapshot to the new region, re-encrypted it with our own KMS key (AWS's key service) and shared it with the other account.

  4. Scripted the copy and the new disk, so a server name was the only input.

  5. Kept the old servers disabled until the new ones had proved themselves.

Problems on the way

  • A snapshot could not be shared with the other account.

    It was encrypted with AWS's default key, which cannot be shared across accounts. Re-encrypted each copy with a key we manage, which can be shared.

  • The old cluster kept all its nodes after the move.

    Nobody had resized it. Summed the pods' memory limits, added 20% headroom and removed three nodes.

Keep the old copy until the new one has proved itself.

Results

  • 1maintenance window for the whole move
  • 7 → 4worker nodes in the old cluster