wiki/clients/current/finwellu/2026-04-05-deployment-mixup-production-restore.md · 1139 words · 2026-04-05

Deployment Mix-up — Production Restore & Workflow Best Practices

Overview

An impromptu call between Mark Hope and Karly Oykhman to resolve a deployment incident affecting the Finwell U website rebuild. A staging-to-production push overwrote the newly built Finwell U site because the developer (Jeff) had built directly on the production environment rather than staging. Mark restored production from a 7:53 PM backup and prepared to switch the finwellu.com domain to point to the recovered site.

Attendees: Mark Hope, Karly Oykhman
Client: [1]


Incident Summary

What Happened

Karly pushed the staging environment to production as part of a standard deployment step. However, because Jeff had built the new Finwell U site directly on the production environment, the push replaced the new site with the old "Financial Wellness Done Right" site that had been sitting on staging. Client Peter was confused to find the old site where the new one should have been.

Root Cause

Jeff built the new site on production rather than staging. He appears to have placed the old site on staging as a reference copy — a workaround that directly caused the overwrite when the staging-to-production push was executed.

Impact


Resolution

Steps Taken

  1. Production restore: Mark restored the production environment to the 7:53 PM daily checkpoint backup, recovering the new Finwell U site.
  2. Verification: Karly confirmed the restored site at nextlevel-prd1.wpenginepowered.com was the correct new build (identifiable by orange design elements).
  3. Domain switch (pending): Mark to set finwellu.com as the primary domain in WP Engine, triggering a URL rewrite across the site.
  4. Redirects (pending): Set up redirects from www.finwellu.com and nextlevel-prd1.wpenginepowered.com to finwellu.com to prevent duplicate content indexing by Google.
  5. SSL fix (pending): SSL certificates to be reset after domain switch completes.

Client Login Fix

Peter's login issue was caused by attempting to use the old domain. Correct login URL is:

finwellu.com/wp-admin

Credentials remain unchanged.


Key Decisions


Action Items


Best Practices Established

This incident prompted a clear articulation of correct WP Engine deployment workflows. See also: [2].

Standard Deployment Workflow

Build on staging → push to production. Never build on production.

Keeping an Old Site Live Alongside a New One

If the old site needs to remain publicly accessible while the new one is being built:

  1. Spin up a new/separate WP Engine server for the old site
  2. Point the old domain (e.g., financialwellnessdoneright.com) to that server
  3. Build the new site on the existing server (staging → production workflow)
  4. When the new site launches, redirect the old domain to the new one via Cloudflare
  5. Decommission the old server when no longer needed

"The old one is just purring along. Do all the building over here. Don't mix apples and oranges." — Mark Hope

Keeping an Old Site for Reference Only (Not Live Traffic)

If the old site only needs to be accessible internally for reference:

  1. Copy it to the development environment in WP Engine
  2. Add a WP Engine site note (under site settings) clearly stating: "DO NOT PUSH — historical reference only"
  3. Communicate explicitly to all team members: do not push the dev environment

⚠️ Risk: The dev environment can still be accidentally pushed to staging, overwriting work. This approach requires clear team communication and is not suitable if the old site needs to serve live traffic.

Why You Can't Run Two Sites on the Same WP Engine Instance

WP Engine environments (production/staging/development) within a single site instance are not independent servers — pushing one overwrites the other. To run two live sites concurrently, they must be on separate server instances with separate domains.

DNS / Domain Concepts (Reference)


Transcript Excerpts

On the root cause:

Mark: "You build in staging, and then when you're done, you send it to production. You don't build in production, and then when you're done, you send it to staging. That's not how you do it."

On the domain vs. site distinction:

Mark: "The URL is not a house. The URL is an address. So there's nothing — that's not the website. The URL is not the website."

On keeping an old site live:

Mark: "If you want to keep the old environment for a while, you create a new server, put that old website on the new server, send traffic to it with the old domain and everything. And then your building over here, it's got nothing to do with the old one."

On the dev environment risk:

Mark: "That development environment thing is kind of fraught with peril too, because somebody can accidentally push the development environment onto the staging site and nuke everything."

On production hygiene:

Mark: "What should be on the production environment? If it's a live website, it should be the live website. If we're building a brand new site, there should be nothing there. It should be empty."

Sources

  1. Index|Finwell U
  2. Wp Engine Staging Production Workflow|Wp Engine Staging → Production Workflow