AH00288 on cPanel: why the Apache scoreboard fills up after graceful restarts

Why "not at MaxRequestWorkers" is the important part

The wording looks contradictory. Administrators normally associate an exhausted Apache scoreboard with the configured worker limit, yet AH00288 explicitly says that the server is not at MaxRequestWorkers.

That distinction matters. MaxRequestWorkers limits how many requests may be served simultaneously, while the scoreboard also has to represent worker and child-process state across Apache generations. A scoreboard can therefore run out of usable slots even when the current generation has not reached the normal request-worker ceiling.

This is why simply increasing MaxRequestWorkers would have addressed the wrong limit in our case.

scoreboard is full, not at MaxRequestWorkers

Graceful restarts keep old and new Apache generations alive together

A graceful restart does not immediately terminate every process from the previous Apache generation. The parent reloads the configuration and starts a new generation, while older children are allowed to finish their current work.

That behavior is intentional. It becomes operationally important when some older workers take much longer than expected to exit. The scoreboard still has to track those old processes while also representing the new generation.

Repeated graceful restarts can therefore create a period in which multiple generations coexist. If enough old slots remain occupied, the scoreboard can become constrained even though MaxRequestWorkers itself has not been reached.

We observed the worker generations overlapping in production

This was not only a theoretical explanation derived from documentation. On the affected systems we reconstructed the process timeline from Apache logs, process start times and graceful-restart events.

A previous generation of Apache child processes remained alive after a later graceful restart had already created another generation. In one observed sequence, workers created around 07:44 were still present after a new graceful restart at approximately 07:50.

That gave us direct production evidence that old and new Apache generations were occupying the server at the same time when AH00288 appeared.

AH00297: SIGUSR1 received. Doing graceful restart

AH00292: Apache/2.4.x configured -- resuming normal operations

• the previous Apache generation,
• the newly started Apache generation,
• and older workers that were still attempting to finish existing connections.

CPANEL-53462 and Apache Bug 61926 appeared in the same investigation

The second important clue was not AH00288 itself but AH00646 from cPanel's splitlogs pipeline.

cPanel documents a known piped-logging race in which Apache graceful restarts can produce Broken pipe errors while splitlogs is in use. cPanel tracks that condition internally as CPANEL-53462 and references Apache Bug 61926 upstream.

That does not prove that the splitlogs race itself caused our scoreboard exhaustion. What it does establish is that another graceful-restart-sensitive subsystem was failing in the same operational window.

AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs: Broken pipe

CPANEL-53462

Apache could be active while HTTPS was effectively unavailable

One of the misleading symptoms was that the service manager still considered Apache healthy.

A basic socket or process check could therefore pass while a real HTTPS request timed out. The Apache parent process had not necessarily crashed; the failure was inside the server's ability to process new web requests correctly.

This is why Apache incident monitoring should not rely exclusively on systemd state or successful TCP connection establishment.

systemctl is-active httpd
active

systemctl status httpd

The incident did not look like CPU or memory exhaustion

We checked the kernel and system logs for the usual resource-exhaustion indicators and did not find evidence that matched the outage.

The affected periods did not show the kind of CPU load that would explain a complete HTTPS failure either. In one incident the load average remained well below 1 on a multiprocessor system while Apache was already failing to serve HTTPS normally.

That allowed us to separate the Apache process-state problem from the much simpler explanation that the server was merely undersized.

• OOM or out-of-memory events
• Killed process messages
• hung tasks
• nf_conntrack table exhaustion
• segmentation faults

Internet scanners were present, but they were not enough to explain it

The servers naturally received substantial automated scanner traffic. We saw probes for environment files, WordPress configuration files, process information, SSH material, Terraform state and cloud credentials.

That traffic can consume connections and expose weak application behavior, but the evidence did not support treating scanner traffic as the complete root cause of this incident.

The stronger correlation was between graceful restart activity, surviving Apache generations, scoreboard exhaustion and splitlogs Broken pipe events.

• /.env
• /wp-config
• /proc/self/environ
• /.ssh/
• Terraform state
• AWS credentials

Why we changed MaxConnectionsPerChild instead of MaxRequestWorkers

The affected worker-MPM configuration used a MaxConnectionsPerChild value of 10000. We increased that value to 50000 on the two systems under investigation.

We deliberately did not raise MaxRequestWorkers. AH00288 was explicitly reporting that the scoreboard was full while the server was not at that limit.

cPanel's own AH00288 support note describes worker-MPM child processes reaching MaxConnectionsPerChild and presents increasing the value as a temporary workaround. That aligned much more closely with the behavior we were seeing.

MaxConnectionsPerChild 10000

MaxConnectionsPerChild 50000

50000 is a mitigation, not a universal Apache fix

The change from 10000 to 50000 should not be copied blindly to every cPanel server.

Our conclusion is narrower: on these systems, increasing MaxConnectionsPerChild reduced how often Apache had to recycle child processes and therefore reduced one source of worker churn around graceful restarts.

The real validation is not that Apache starts successfully after the change. The meaningful test is whether the server remains healthy through future automatic graceful restart cycles without returning to AH00288.

The cPanel configuration layer matters

We did not treat the generated Apache configuration file as the authoritative place to make a permanent cPanel change.

On older EasyApache 4 configuration interfaces, the internal attribute may still appear under the historical name maxrequestsperchild even though the generated modern Apache directive is MaxConnectionsPerChild.

That distinction is important when automating or auditing cPanel configuration because the control-panel representation and the resulting Apache syntax do not necessarily use the same name.

maxconnectionsperchild

maxrequestsperchild

/etc/apache2/conf/httpd.conf

We performed a clean Apache stop and start after changing the value

After changing the configuration through cPanel's configuration layer, we did not rely only on another graceful restart.

The objective was to remove the existing old worker generations and begin testing from a clean Apache process state. Otherwise the same surviving children that were part of the incident could have remained visible after the configuration change.

Imunify360 was a restart trigger in our logs, not a proven root cause

The production logs showed that Imunify360-related activity could initiate web-server graceful restarts in the observed environment.

That correlation is operationally relevant because repeated graceful restarts are central to the worker-generation overlap we were investigating.

However, we are not claiming that Imunify360 causes AH00288. It was one observed source of graceful restart activity, not a demonstrated root cause of the scoreboard problem.

Performing web server graceful restart

create_rbl_whitelist

update_vendors

Why we did not simply disable cPanel splitlogs

cPanel documents disabling piped logging as a workaround for the splitlogs Broken pipe condition, but that was not our first choice.

The piped logging architecture exists for a reason on hosting systems with many domains: it avoids Apache directly holding a large number of individual log files open.

Because AH00646 was supporting evidence rather than a proven root cause of AH00288, removing splitlogs would have changed another significant part of the hosting architecture without proving that the scoreboard issue had actually been solved.

What we monitor after the mitigation

A successful restart immediately after a configuration change is not enough. The systems have to survive the same type of automatic graceful-restart activity that was present before the mitigation.

For that reason the next validation window focuses on both availability and process behavior.

• Apache service state
• real HTTPS responses rather than only TCP connectivity
• start times of newly created Apache PIDs
• splitlogs process restarts
• the generated MaxConnectionsPerChild value
• new AH00288 and AH00646 events

Event MPM is relevant, but we did not change MPM during this incident

Apache's event MPM documentation explicitly discusses the situation in which old processes remain in Graceful state after a restart and continue consuming scoreboard capacity.

That makes event MPM highly relevant to understanding the mechanism behind this class of warning.

We did not switch the affected production systems from worker MPM to event MPM as part of this mitigation. Changing the MPM would introduce another major variable before the existing evidence had been validated.

A practical diagnostic sequence for AH00288 on cPanel

When AH00288 appears on a cPanel server, the error should not automatically be treated as a request-volume problem.

The first goal is to determine whether the scoreboard is constrained by current workload or by old Apache generations that remain alive across graceful restarts.

• confirm whether the log says not at MaxRequestWorkers
• correlate AH00288 with AH00297 graceful restart events
• check whether older Apache child processes survive into later generations
• look for AH00646 and splitlogs Broken pipe errors in the same window
• verify actual HTTPS availability, not only systemd or socket state
• check CPU, memory, kernel and connection-table evidence before assuming resource exhaustion
• verify the generated MaxConnectionsPerChild value after any cPanel-side change

What this incident proves, and what it does not

The strongest conclusion from this investigation is that the observed AH00288 events were associated with overlapping Apache worker generations after graceful restarts, while the server remained below MaxRequestWorkers.

The simultaneous splitlogs Broken pipe condition is consistent with cPanel's CPANEL-53462 documentation and Apache Bug 61926, giving us an additional graceful-restart-related signal from the same period.

Increasing MaxConnectionsPerChild from 10000 to 50000 is therefore being treated as a mitigation that reduces child-process recycling pressure. It is not being presented as a universal fix, and future automatic graceful restarts remain the real test.

Official documentation and upstream references

• cPanel: AH00288 scoreboard is full, not at MaxRequestWorkers
• cPanel: splitlogs Broken pipe during Apache restarts
• Apache Bug 61926
• Apache documentation: stopping and restarting
• Apache event MPM documentation
• cPanel Piped Log Configuration