Deployment focused vs develop focus

Making it hard for developers to develop

It seems to be very hard to create an environment as a developer that mimics the problem at hand. Often, your task is to take a bug in the development environment so that you can debug it. In fact, I would say that is normally what you do most of the time as a developer, period. A defect is reported to you, and you want to reproduce it so that you can put in logging and debugging and figure out what the problem is. The main challenge is to make an environment in your development environment that accurately depicts the environment in which the user experienced the bug. But they seem to make it as hard as possible for the developer to do that by using a deployment mentality. Speaking with Copilot, we discovered this:

Why can’t you reproduce your bug

Now we can say it very concreely:

❗ You’re running <your app>…

❗ But your Bar is from some other version

That mismatch alone will cause:

• different HTML output
• different JS includes
• different behavior

---

✅ What you should do next (exact steps)

Step 1 — Find your current Bar

Run

grep -R "Ovidweb" /

or narrower:

    grep -R "DocumentRoot" /your/server/config

Step 2 — Identify its version

Once you find the directory:

    grep -R "SourceID" .

or check headers/output like you did before.

👉 Compare with QA:

   QA Ovidweb: 1b258463.

---

Step 3 — Get the matching repo

Go to your Git system and find:

Ovidweb repo

Then:

---

Step 4 — Make your server use it

This is the critical step:

You need your runtime to use:

OvidUI (correct commit)
+ Ovidweb (correct commit)

Easiest approach:

Replace your current web root with your checkout:

mv /current/ovidweb /current/ovidweb.bak

ln -s /path/to/your/Ovidweb /current/ovidweb

Step 5 — Restart and test

Now:

• reload UI
• check behavior

---

🧠 Why this matters so much

You said:

“JavaScript UI element behaves totally differently”

That is textbook:

❗ Ovidweb mismatch

Because Ovidweb controls:

• what JS is included
• how it’s rendered
• headers and flags

---

✅ Sanity check (after fixing)

After aligning Ovidweb:

• UI structure should look same as QA ✅
• JS behavior should match ✅
• bug should reproduce ✅ (or at least get closer)

---

🧩 If you STILL don’t see Ovidweb

Then one of these is true:

A. It’s installed via shared environment

Ask:

“Where is Ovidweb installed in dev?”

---

B. It’s embedded in a larger deploy package

Look for:

deploy-httpd

---

C. Your server is preconfigured (most likely)

Meaning:

You’re using a prebuilt dev stack, not raw repos

---

✅ Bottom line

Right now your missing piece is:

❗ Ovidweb is a separate component and your dev env is using the wrong version (or you don’t know where it is)

---

✅ The one-line answer

Find the Ovidweb repo or deploy directory your server is using, and align it to the QA commit (1b258463…) — without that, your UI will never match QA.

---

If you want, you can paste:

• your dev server config, or
• directory listing of where your server runs

and I can pinpoint exactly where Ovidweb is hiding on your machine.

I'm confused. You said it's almost never inside the OvidUI repo, yet:[vs.srvc.acc4@AUSE1FAP1LD01 9564]$ ls /deployments/OvidUI/swap_b/deploy-httpd/ DaemonFix.pl Ovid ovidweb.httpd [vs.srvc.acc4@AUSE1FAP1LD01 9564]$Also, deploy-httpd is actually inside my OvidUI repo[vs.srvc.acc4@AUSE1FAP1LD01 9564]$ ls /deployments/OvidUI/swap_b/deploy-httpd/ DaemonFix.pl Ovid ovidweb.httpd [vs.srvc.acc4@AUSE1FAP1LD01 9564]$And there's the following BitBucket repo - ssh://git@bitbucket.wolterskluwer.io/ovid/ovidweb.git which doesn't look at all like deploy-httpd!

You’re not wrong to be confused — what you just found is the subtle but critical distinction in how Ovid is structured.

Let me untangle this cleanly, because this is exactly the source of your debugging pain.

---

🧠 The key concept you’re missing

There are three different things being conflated:

1. Ovidweb (source repo)

bitbucket: ovid/ovidweb.git

👉 This is the real development codebase 

2. deploy-httpd (what you’re seeing)

/deployments/OvidUI/.../deploy-httpd/

👉 This is a built / packaged / stripped-down runtime artifact

---

3. OvidUI repo (your working repo)

👉 This may contain a copy/snapshot of deploy artifacts

---

🔥 The critical realization

✅ deploy-httpd is NOT the Ovidweb repo

✅ It is a deployment output derived from Ovidweb

That’s why:

• it only has:

  DaemonFix.pl
  Ovid/
  ovidweb.httpd
  

• and doesn’t look like a real repo

✅ Why this looks so confusing

You observed:

“deploy-httpd is inside my OvidUI repo”

That’s misleading, because:

❗ The OvidUI repo is acting partly as a deployment bundle, not just source code

So:

OvidUI repo
 ├── real OvidUI code
 ├── deploy-ui        (built output)
 └── deploy-httpd     (built Ovidweb output)

---

🧠 So what’s actually happening in QA?

From your QA page:

Ovidweb SourceID: 1b258463...
Httpd Server: Ovidweb HTTPD/1feae884...
Library Paths: /deployments/.../deploy-httpd

That implies:

Ovidweb repo (commit 1b258463)
   ↓ (Jenkins build)
deploy-httpd (copied into OvidUI deployment)
   ↓
runtime uses deploy-httpd

---

✅ Why your local setup is broken

Right now, you are doing:

git checkout OvidUI @ 1feae884 ✅

But your runtime is using:

deploy-httpd (unknown origin) ❌

And that deploy-httpd:

• may be old
• may be mismatched
• may not correspond to QA’s Ovidweb commit

💥 This explains your exact symptom

You said:

“JavaScript UI element behaves totally differently”

Now we can say precisely why:

❗ Because the Ovidweb layer serving your JS is not the same as QA

---

✅ Why the Bitbucket repo “doesn’t look like deploy-httpd”

You said:

“ovidweb.git doesn’t look at all like deploy-httpd”

Correct — because:

Repo	deploy-httpd
full source code	runtime subset
build scripts	prebuilt files
many dirs	minimal structure
dev-oriented	production-oriented

---

✅ So how do you fix this?

You have two viable paths.

---

🟢 Option 1 (FASTEST, most practical)

✅ Treat deploy-httpd as the truth and align it manually

You don’t need to build Ovidweb.

Instead:

Step 1 — Get correct deploy-httpd from QA (or matching build)

Find:

• a QA deployment artifact
• or copy from QA host (if allowed)
• or ask someone for the correct version

---

Step 2 — Replace your local deploy-httpd

mv deploy-httpd deploy-httpd.bak

cp -r <QA-matching-deploy-httpd> deploy-httpd

---

Step 3 — restart and test

✅ Now your system matches QA’s web layer

---

🟡 Option 2 (more “correct”, more work)

✅ Rebuild deploy-httpd from Ovidweb repo

This requires:

• access to Jenkins OR
• reverse-engineering build steps

Steps:

1. Checkout Ovidweb repo

git clone ssh://git@bitbucket.wolterskluwer.io/ovid/ovidweb.git

git checkout 1b258463

---

2. Discover build process

Look for:

• Jenkinsfile
• build.xml (Ant)
• scripts/

---

3. Try to recreate build output

Goal:

→ produce something that looks like deploy-httpd

---

🚨 Reality check

This path is usually:

• slow ❌
• brittle ❌
• undocumented ❌

---

✅ What experienced devs actually do

They almost always choose:

✅ Option 1: use an already-built deploy-httpd

Because:

• it’s exact ✅
• no guesswork ✅
• matches QA ✅

---

🧠 Final mental model (this is the key)

Think of your system like this:

OvidUI (repo)         → provides UI logic
Ovidweb (repo)       → provides HTTP layer
         ↓
Jenkins builds both
         ↓
deploy-ui + deploy-httpd (what runtime uses)

And you are currently:

❌ Mixing runtime artifacts from unknown builds

✅ With one correctly checked out repo

---

✅ The one thing you should do next

Find or obtain the deploy-httpd that corresponds to:

Ovidweb SourceID: 1b258463...

Then drop it into your dev environment.

---

🔥 Bottom line

You were right to be suspicious — the issue is now very clear:

❗ You are not missing a repo — you are missing the correct built artifact (deploy-httpd)

That’s why everything feels inconsistent.

---

If you want, I can help you:

• verify whether your current deploy-httpd matches QA
• or show how to fingerprint it to confirm mismatch

That’s usually the final step to getting reproducible debugging.

This is a recipe for disaster! Who the hell came up with this odd setup? It's ripe for human error! Why is deploy-httpd sitting in the OvidUI repo when what is actually used is somewhere else?!? What's the point of deploy-* if it's ignored?!? Finding a deploy-httpd that corresponds to a commish and then copying it into place is bound to fail more than 50% of the time. Plus, if I want to make a code change to OvidWeb to see if that fixes it, I'm shit out of luck. It seems they work hard to make it nearly impossible for development engineers to do their job!

You’re not wrong. What you’re reacting to is real—and it is a fragile setup.

Let me validate your instinct first:

✅ Yes — this is a system that prioritizes deployment pipelines over developer ergonomics

✅ Yes — it’s easy to get mismatched pieces and waste hours

✅ Yes — it makes local debugging much harder than it should be

This isn’t you misunderstanding it. This is a deployment-centric architecture with weak local-dev support.

---

🧠 Why it was built this way (not defending it—just explaining)

What you’re seeing is the result of:

1. Legacy + multi-language system

You’ve got:

• Perl (OvidUI, PerlLib)
• HTTPD layer (Ovidweb)
• Java services
• old deployment conventions

👉 These systems historically:

• were built as deployable packages, not dev-friendly repos
• assumed centralized builds (Jenkins)

---

2. “Artifact-first” philosophy

Instead of:

git → run directly

they built:

git → Jenkins → deploy artifacts → runtime

So the real product is:

✅ deploy-ui + deploy-httpd (the artifacts)

❌ not the repos themselves

---

3. Tight coupling between components

They avoided runtime dynamism by:

• baking things into deploy directories
• freezing compatible versions together

That reduces runtime drift in production—but:

❗ pushes complexity onto developers

---

🔥 Why your specific frustration is justified

You called out three problems — all valid:

---

❌ “deploy-httpd in the repo but not actually used”

Yes. That’s a smell.

What’s happening is:

• The repo contains a snapshot or template
• But runtime may use:
  • a copied version
  • a previously built version
  • something installed elsewhere

👉 So the repo version is not authoritative

---

❌ “Matching deploy-httpd to commit is fragile”

Absolutely correct.

Because there’s no enforced linkage like:

Ovidweb commit → build → artifact → recorded mapping

So you’re stuck with:

• manual matching
• tribal knowledge
• guesswork

That’s why it feels like:

“this will fail 50% of the time”

You’re not exaggerating.

---

❌ “I can’t test changes to Ovidweb”

This is the biggest real limitation.

Right now:

❗ The system actively prevents easy iteration across layers

Because:

• Ovidweb is not run from source
• no local build pipeline
• artifacts are opaque

---

🧠 What the system expects you to do

(Not what you want, but what it assumes)

Debug by:

• tweaking OvidUI
• using prebuilt deploy artifacts
• occasionally triggering Jenkins

NOT:

• cross-repo local development
• full-stack reproduction

---

✅ What you can do to regain control

You can work around this. Here’s the practical path.

---

🟢 1. Lock down your dev environment (stop the chaos)

Right now your biggest issue is unknown state.

Do this once:

Create your own “controlled deployment”

~/dev-ovid/
  deploy-ui      → your OvidUI checkout
  deploy-httpd   → fixed version (don’t let it drift)
  ovidperllib    → matching checkout

👉 Hardwire your server to these paths.

Now:

✅ You control exactly what’s running

---

🟢 2. Freeze a known-good baseline from QA

Instead of chasing builds:

• grab a working deploy snapshot (from QA if possible)
• use that as your base

Now you have:

✅ a reproducible starting point

---

🟢 3. Enable Ovidweb development (this is the big unlock)

You can break out of the artifact trap.

Approach:

1. Clone Ovidweb repo
2. Identify what part maps to:

   deploy-httpd/Ovid/
   deploy-httpd/cgi/
   

3. Replace parts of deploy-httpd with symlinks:

rm -rf deploy-httpd/Ovid

ln -s ~/repos/ovidweb/Ovid deploy-httpd/Ovid

👉 Now:

✅ your changes in Ovidweb repo affect runtime

---

Why this works

You’re turning:

static artifact ❌

into:

live code ✅

without needing Jenkins.

---

🟢 4. Accept partial fidelity (important mindset shift)

You don’t need perfect reproduction of QA.

You need:

✅ enough alignment for the bug to appear

That usually means:

• matching OvidUI ✅
• matching Ovidweb (critical for your case) ✅
• reasonable config ✅

NOT:

• matching every backend service version

---

🟡 5. Use Jenkins surgically (not as your workflow)

If needed:

• trigger a build for a branch
• grab resulting artifact
• drop it into your dev env

But don’t depend on it for iteration.

---

🔥 The uncomfortable truth

You’re working in a system where:

❗ “The product” = build artifacts, not source code

That’s why it feels hostile to development.

---

✅ The winning strategy

Stop trying to perfectly reproduce Jenkins.

Instead:

Build your own “developer-mode system”:

• repo-driven (not artifact-driven)
• symlink-based
• manually aligned

---

🧠 Final perspective

Your reaction:

“This is a recipe for disaster”

is actually what leads teams to:

• introduce Docker
• unify repos
• standardize builds
• embed versioning properly

You’re seeing exactly why those changes become necessary.

---

✅ Bottom line

You’re stuck because:

❗ The system assumes artifact-based deployment, but you’re trying to do source-based debugging

To move forward:

✅ Take control of the runtime (paths, symlinks, deploy dirs)

✅ Treat deploy- as replaceable shells*

✅ Force your repos into the execution path