Onboarding customer files to Cloudflare R2
A customer's files live on the server's own diskits own R2 bucket.
I led the cloud side. Product engineers built the sync itself.
Customers store filesets, the software and files they push to their devices, on their server. Each server is a single pod on Amazon EKS.
Why it was needed
Filesets lived on each pod's own disk. A new release could store them in Cloudflare R2, but the first upload blocks the server while it runs, and EU customers' files had to stay in the EU.
How it was done
Added an init container (a container that must finish before the server starts) that runs the first upload and keeps the server down if it fails.
Rolled the new chart out to every production cluster with the upload switched off.
Built sync metrics and a dashboard in Terraform, so each upload has proof.
Added a jurisdiction option that keeps a bucket's data in the EU, because a bucket cannot change it later.
Problems on the way
New servers still got the old settings.
The server-creation script wrote the old keys. Fix: Changed it to write the new settings, with everything off.
A sync hung after a pod stop.
A stale progress entry in the sync's message store said a sync was still running. Fix: Deleted the entry and restarted the pod.
Ship the chart with the feature off, then turn it on one customer at a time.
Results
- 4production clusters on the new chart, upload off
- 1maintenance window for the EU switch
- 0leftovers after the switch