Migration scripting is the practice of writing deterministic, repeatable scripts that extract, transform, and reconcile state between computing environments. It matters when manual processes become error-prone across dozens of clusters, when you’re synchronizing configurations after a platform upgrade, when you’re migrating database schemas between engine versions, or when you’re automating the transfer of infrastructure-as-code between version-controlled stores and live systems. This page gives a concrete, tool-level workflow: what to run, what flags matter, how to validate, and the failure modes that catch out even experienced teams.

1. Extracting the Source State

The first step in any migration script is a faithful, lossless export of what exists today. The tooling depends on the layer, but the pattern is always the same: render the current desired state to a portable format, then script against it.

  • Kubernetes manifests – Render the live cluster into YAML:
  kubectl get all,configmaps,secrets -A -o yaml > cluster-state.yaml

The -A flag namespaces all namespaces; omit it to stay within a single namespace.

  • PostgreSQL schemas – Dump a binary, self-contained backup that includes data and definitions:
  pg_dump --schema=public -Fc -f pre-migration.dump

The -Fc (custom format) flag enables parallel restoration with pg_restore -j N and preserves object identifiers, which is important when you need to map old OIDs to new ones during transformation.

  • Docker Compose files – Export the running service graph:
  docker compose config > compose-export.yaml

This writes the resolved, variable-substituted file to stdout; redirect it to a path you control.

  • Git history – When the “source of truth” is version-controlled, start from the last known good commit:
  git log --diff-filter=M --name-only --format="%H %s" v1.2..v1.3 > changelog.txt

If the source is an API, use curl with pagination flags to walk every page:

curl -sS "https://api.example.com/resources?page=1&per_page=100" > page1.json

2. Transforming the Data

Once the source state is on disk, the script’s job is to rewrite it into the target shape. Keep the transformation pure: read, mutate, write. Avoid side effects like kubectl apply inside the parsing phase.

A typical Python pattern using PyYAML and yaml merging:

#!/usr/bin/env python3
import sys
import yaml

def bump_image_tag(path):
    with open(path) as f:
        manifest = yaml.safe_load(f)

    for container in manifest.get("spec", {}).get("containers", []):
        image = container.get("image", "")
        if ":" in image:
            name, tag = image.rsplit(":", 1)
            container["image"] = f"{name}:{str(int(tag) + 1).zfill(3)}"

    with open(path, "w") as f:
        yaml.dump(manifest, f, sort_keys=False)

if __name__ == "__main__":
    for filepath in sys.argv[1:]:
        bump_image_tag(filepath)

If you prefer in-place stream editing, yq (the YAML processor) and jq (the JSON processor) are standard, battle-tested:

  • Raise every image tag by one minor version across a directory of manifests:
  yq eval '.spec.containers[].image |= (split(":") | .[0] + ":" + (.[1] | tonumber + 1 | tostring))' -i k8s/**

The -i flag edits files in place; without it, the command prints the transformed tree to stdout.

  • Pull out every Deployment resource that has replicas: 0 and set it to the default 1:
  jq '.. | objects | select(.kind == "Deployment" and .spec.replicas == 0) | .spec.replicas = 1' input.json > output.json
  • Replace a config map’s binary data with a base64-encoded version of a local file:
  yq eval '.data[ "certs.pem" ] = (input("certs.pem") | @base64)' cluster-state.yaml

When the target format is JSON rather than YAML, the same logic applies with jq. Keep a transform.log file that records each substitution, the old value, and the new value—this log becomes your audit trail and the primary source for rollback analysis.

3. Applying to the Target

With the transformed file ready, the next phase is controlled application. Always test the apply command in dry-run mode before writing to the live stack.

  • Kubernetes – Dry‑run and validate (but don’t enforce) the manifest:
  kubectl apply -f cluster-state.yaml --dry-run=client --validate=false

--dry-run=client checks the local file against the Kubernetes API server’s schema without sending anything. --validate=false skips the server-side schema validation, which is useful when the target cluster runs a different minor version than the workstation. If you need full server-side validation, drop the --validate=false flag and ensure the API versions match.

  • Apply for real:
  kubectl apply -f cluster-state.yaml

This command is idempotent: if the manifest already matches the live state, kubectl apply reports “resource is unchanged.” If the manifest differs, it patches or creates the resource. For immutable resources (Deployments, ConfigMaps), it performs a strategic merge patch.

  • PostgreSQL – Restore the dump into a fresh database:
  pg_restore -d target_db -j 4 pre-migration.dump

The -j 4 flag runs four parallel workers, which on a modern SSD cuts a 500 MB dump from ~45 s to ~12 s. If you’re migrating into a point‑in-time recovery window, pipe through psql with a SET enable_indexscan = off directive to force sequential scans during the restore.

  • Docker Compose – Recreate services in the target environment:
  docker compose up -d --remove-orphans

The --remove-orphans flag strips containers that no longer appear in the compose file, preventing ghost processes from lingering after a migration.

  • Git‑driven infrastructure – If you’re using a GitOps loop (e.g., Argo CD, Flux), the apply step is already handled by the controller. Your script’s job ends at committing the transformed file:
  git add k8s/cluster-state.yaml
  git commit -m "chore: bump image tags across all deployments"
  git push origin main

The GitOps controller then syncs the desired state to the cluster automatically.

4. Validation and Verification

After the apply step, you need to prove that the live state matches the intended state. Compare, don’t guess.

  • Kubernetes – Diff the live inventory against the file you just applied:
  kubectl get all,configmaps,secrets -A -o yaml > live-state.yaml
  diff -u cluster-state.yaml live-state.yaml | head -n 80

If the diff is empty (aside from metadata timestamps), the migration succeeded. Any remaining lines indicate resources that either failed to apply or required manual adjustment.

  • PostgreSQL – Compare row counts and checksums between source and target:
  psql -c "SELECT count(*) FROM pg_tables WHERE schemaname='public';" target_db
  pg_dump -t old_table -Fc -f after-migration.dump target_db
  pg_dump -t old_table -Fc -f before-migration.dump source_db

Then run pg_diff (part of the pgformatter suite) or manually pg_restore --check to surface OID mismatches.

  • Docker Compose – List running services and their desired state:
  docker compose ps

Compare the DESIRED STATE column (Created/Running/Stopped) against the docker-compose.yml you just applied.

  • Git – Verify the commit that introduced the change is the one currently checked out on the GitOps controller:
  git log --oneline -1

5. Rollback and Recovery

Even well-tested scripts can hit an unexpected drift. Know how to undo before you start.

  • Kubernetes – Undo the last rollout:
  kubectl rollout undo deployment/analytics-service

This sets the Deployment’s revision back to the previous replica set. kubectl rollout status deployment/analytics-service will confirm the rollback completed.

  • Kubernetes – Revert a ConfigMap or Secret to its previous value:
  kubectl replace -f <(kubectl get configmap my-config -o yaml | yq eval 'del(.metadata.resourceVersion, .metadata.UID) | .data["old-key"] = "old-value"')

The resourceVersion and UID must be stripped; otherwise kubectl replace treats the object as a new resource and creates a duplicate.

  • PostgreSQL – Restore from the pre‑migration dump:
  pg_restore -d source_db -j 2 pre-migration.dump

If the migration altered schema objects (added columns, dropped tables), you’ll need to run the migration scripts in reverse order, or use pg_dump --schema=public -s to script-only the definitions and psql -f them back.

  • Git – Hard‑reset to the commit before the migration script ran:
  git reset --hard HEAD~1
  git push --force origin main

This rewrites the remote history; coordinate with your team if the branch is shared.

Sharp Edges — What Silently Fails, What Catches You Later

  • Label selector drift – A migration script that renames a Deployment’s metadata.labels field without updating the spec.selector.matchLabels will cause the Deployment to stop managing its Pods. The old Pods continue running, but new replicas are never created. The symptom: “Deployment stuck at 1/2 available.” The fix: always reconcile selector fields in the same atomic step as the label change.
  • --dry-run=client vs --dry-run=server – The client‑side flag validates against the local kubectl version’s schema; the server‑side flag hits the API server. If your workstation runs v1.27 and the cluster is v1.25, the client may accept a field the server rejects, or vice‑versa. Always run both: kubectl apply --dry-run=client --dry-run=server -f file.yaml. If they disagree, inspect the Warning field in the output.
  • pg_dump custom format parallelism – The -j flag parallelizes the *restore*, not the dump. Dumping a 2 TB database with -j 8 on the dump side does nothing; the bottleneck is usually disk I/O on the source. Parallelize the restore into an empty target, or use pg_basebackup + pg_replicate for massive migrations.
  • Docker Compose variable resolution order – docker compose config resolves $VAR environment variables, then .env files, then variables: blocks in the compose file, in that order. If your migration script injects a new env var but the compose file also defines it under variables:, the explicit environment: key wins. Document which source takes precedence, or the rebuilt services will start with the wrong credentials.
  • Idempotency pitfalls – kubectl apply is idempotent for *declarative* resources, but not for *imperative* ones. If your script does kubectl create deployment foo --image=bar inside a loop, the second iteration throws “deployment foo already exists.” Switch to kubectl apply -f - with a manifest, or use kubectl patch for in‑place updates.
  • Silent secret truncation – Some migration scripts base64‑encode secret data, pipe it through kubectl create secret generic, and never check the return code. If the base64 string exceeds 1 MiB (the Kubernetes limit for secret data values), the API returns 200 OK but the secret is stored with truncated content. The Pod then fails to start with “secret too short.” Always kubectl get secret name -o jsonpath='{.data.key}' | wc -c after creation to verify length.
  • GitOps sync loops – If your migration script commits a manifest that changes a resource’s spec.replicas from 3 to 0, the GitOps controller may immediately scale the Deployment to zero, then your script’s next run reads “0 replicas” and tries to set it back to 3, creating a rapid sync loop. Guard against this by either pausing the controller during migration (argocd app pause <name>) or by using a temporary “dry‑run” branch that you merge only after verification.
  • Time‑zone–sensitive data – When migrating logs or timestamps between systems, always normalize to UTC before scripting. A naive SELECT created_at FROM orders can return different wall‑clock times depending on the server’s TimeZone setting, leading to duplicate or missing records in the target. Explicitly SET timezone = 'UTC'; at the top of every migration SQL file.
  • Missing terminationGracePeriodSeconds – When migrating Pods that handle graceful shutdowns (e.g., web servers finishing in‑flight requests), a script that only kubectl delete pod without --grace-period=30 can kill processes mid‑request. The Pod enters Terminating but the OS sends SIGKILL after the grace period expires. Always include the flag, or set it as a default in your cluster’s PodSecurityPolicy/AdmissionController.
  • yq version drift – The yq command’s API changed between the mikefarah/yq (v4) and kislyuk/yq (v3) branches. yq eval '.spec.containers[].image' file.yaml works in v4 but throws “syntax error” in v3. Pin the version in your CI matrix: brew install mikefarah/yq/yq@4 or apt-get install yq pulls the distributor‑specific version. Add a yq --version check at the top of your migration script and abort with a clear message if the major version doesn’t match the one documented in your runbook.
  • kubectl auto‑completion side‑effects – Some organizations ship kubectl with custom plugins that add subcommands like kubectl migrate. If your script shebang #!/usr/bin/env bash sources /etc/bash.bashrc and that file loads a plugin that redefines apply, your script may silently call the plugin’s version instead of the upstream binary. Use the full path /usr/bin/kubectl in scripts, or command kubectl to bypass shell functions.
  • Dry‑run‑client does not validate secrets – kubectl apply --dry-run=client -f secret.yaml succeeds even if the secret’s data field contains non‑base64 text. The server will reject it with BadRequest, but the client‑side dry‑run lies. If you’re migrating secrets, always follow the dry‑run with kubectl apply --dry-run=client -f secret.yaml 2>&1 | grep -i "bad request" or simply apply to a test namespace first.

When you treat migration scripting as a series of deterministic, introspectable commands—extract with kubectl get / pg_dump, transform with yq/jq, validate with diff / pg_restore --check, apply with kubectl apply / pg_restore, and roll back with kubectl rollout undo / git reset—you eliminate the “it worked on my machine” class of failures. The sharp edges listed above are the ones that surface in production at 2 AM; keeping them in your runbook means the difference between a controlled cutover and a multi‑hour incident.

This page was rewritten on 14 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.