How to waste time chasing ghosts

Wasted a few days chasing ghosts

During my development workflow, I spend time running our server process after making code changes. Often, a problem will happen with some JavaScript caching or umask-induced permissions problem. The result is that you're making code changes and not seeing any change.

But caching and permissions problems are not the only way this can happen. It can also happen by having to restart the server process when the control script does not properly handle things, resulting in multiple server processes. Then the code you just wrote does not appear to work because you are not talking to the server process you think you're talking to.

This can happen because:

1. Multiple server processes can share the same port
2. Handling of starting, stopping, and cleaning up is not done correctly
3. Process is more complex due to the insistance of using "shared user  accounts"

Two separate things were happening.

First, multiple processes per “one server” is expected here. In httpd.conf:20, prefork=1 means this httpd starts as a small prefork server, so each launch gives you:

1. One master process
2. One worker process

That’s why the healthy current instance shows a pair like 4040369 and 4040378.

Second, the reason you accumulated many old pairs is the control script is weak about lifecycle management. In owhttpdctl:21, it trusts only one pid file, httpd.conf.pid. On stop, it only does:

1. kill -TERM $PID
2. Deletes the pid file

It does not:

1. Verify the PID is still the right live master
2. Clean up stray worker processes
3. Refuse restart when old processes still exist but the pid file is stale
4. Kill by port or by process group

So the failure mode is:

1. A server pair gets started
2. The pid file becomes stale or out of sync
3. stop kills the wrong PID or a dead PID
4. The old master/worker pair survives
5. Another start creates a new pair
6. Repeat over days, and you end up with many orphaned sleeping ovidweb.httpd processes under your user

The key evidence is:

1. Old process pairs date back to Jul 2, Jul 3, Jul 8, Jul 9, Jul 10
2. Their PPID is 1, which means they were orphaned/reparented
3. Only the newest pair was actually bound to port 9564 when I checked with ss

So: multiple processes for your user happened because each launch creates a pair, and stale pid-file-based stop/start handling let old pairs accumulate instead of being cleanly replaced.