Zogograph infrastructure observation dashboard

There are forty pages here, each answering one question well — which helps when you already know the question. This reads all of them at once. Every line carries the measurement it came from and a link to the page that shows it in full, and nothing here is inferred from a correlation.

On a shared host the numbers above are the cage's answers. Inside CageFS this account is shown a /proc written for it: its CPU allowance looks like the machine's cores and its memory limit looks like the machine's RAM. Comparing both readings is the only way to tell a limit from the size of the server.

The logs named in host_log_paths. These belong to root and live outside this account, so on most shared servers they will be listed and refused — which is itself the answer. Where the host has granted read access, they open in the same viewer as the account's own logs.

Names are hidden until you sign in. The sizes and the growth are real.

Area is size. A directory full of small files is invisible in a list of the biggest paths and obvious here, because forty thousand session files add up to a large rectangle even though not one of them is large. Click a rectangle for its details, double-click a directory to go into it, hover for a quick summary. Filtering to one type shrinks every directory to just the files of that kind inside it, which is how you find where the audio is.

Backup storage

Backups are usually the largest thing on a server and the last place anybody looks, because they are not in /home. This asks four questions: how much space, whose, how old, and what is safe to do about it.

What can be done about it

Sorted by how safe each one is, not by how much it would free. Anything marked destructive removes the only copy of something — the whole point of a backup is that it is there on the day nobody expected to need it.

Other probable backups

Directories that look like backups but are not managed by anything: a tar.gz somebody made before an upgrade, a dated copy of a site, a database dump. Nothing rotates these, so they stay for ever and are usually the oldest large thing on the disk. Probable is the operative word — each is a guess from the name and shape, and worth looking at before acting.

JetBackup

What the backup system says about its own backups, which is the only place the account and date behind a directory called files0000000005 can be found. It needs the JetBackup command line tool and usually needs running as root.

From host-cron.php, run by root. Inside CageFS this account gets its own /proc holding only its own processes, so everything below is invisible to every other panel on this site. The detail sits on the pages beneath this one; what is here is the state of the machine's memory at a glance. Its disks are under Storage, and whether its daemons are alive is under Processes → Services.

The column CageFS hides. Inside a cage this account sees only its own processes, so who else is using the machine's memory is unanswerable from anywhere but here.

One line per service, from the host collector's own longer-lived record. This is the panel that answers “and what was it in June?”

The share of measured process memory each service holds, drawn from PSS rather than RSS — a pie claims its slices make a whole, and resident memory cannot: a page shared by ten processes would be counted ten times.

Every process on the machine, not just this account's. This is the list that answers “what is actually holding the memory” when the per-user totals only say whose it is.

Memory the kernel holds for itself. It belongs to no process and no account, and it is the first place to look when the numbers do not add up — a quarter of a gigabyte of xfs_inode appears in nothing else on this site.

Throughput and saturation per device, read by root — so this is every account on the server together. A busy line here says the disk is working, never that this account is the reason; the account's own traffic is on the dashboard, and which processes are doing it has its own page.

Filesystems, and the space nothing accounts for

For when df reports a number nothing else can explain. Every filesystem root can see, and the gap between what the filesystem says is used and what walking it actually found. The two usual explanations have pages of their own: deleted files still held open and the biggest directories.

Deleted, still holding space

A file that has been unlinked while a process still has it open keeps its blocks. df counts them; du, ls, find and the tree map cannot see them at all. This is the one kind of missing space no directory walk can ever find, and restarting the process listed is what frees it.

Biggest directories on the machine

From root's walk of the whole disk, so this reaches the places the account cannot: other accounts' homes, /var, /backup, and anything mounted somewhere unexpected.

Which processes the throughput is actually coming from. CPU can sit low while one job makes everything else feel slow by keeping a disk busy, and that job is invisible in every other panel on this site.

Apache writes each domain's logs under /etc owned by root. A rotated log there costs exactly as much disk as one in the home directory and is invisible to everything the account can run.

Every mount the kernel reports, grouped by filesystem. Two paths on the same device are the same disk however different they look, which is why a bind mount can make one directory appear to be two — and why summing mount points reports the same space twice. The skeleton's own mounts used to be the second half of this page and are now on the CageFS skeleton page, which is where somebody looking for them goes.

CageFS gives each account its own /usr, /lib and /proc, assembled from a shared skeleton at /usr/share/cagefs-skeleton. That is why a program the server has can be missing from inside the account, and why this dashboard's view of the machine is narrower than the machine. From inside a cage the skeleton's own mounts are invisible — so an empty answer here is not evidence that CageFS is absent, and this says which of the two it is looking at.

Running right now, with processes of the same name added together. “Running for” is the oldest of the group — a long-lived pool whose newest member is seconds old is restarting as fast as it dies. Arguments that look like passwords are hidden.

Reported process load over this range

The newest reported processes lead this list, with high peaks retained so a process that has settled down is still visible. Names group replacement workers together; PIDs are intentionally not a chart identity.

Stop by command line

For when the name is not specific enough to aim with — twelve webalizer processes where one of them is chewing through one particular log. This is pkill -f, with the two differences that matter: the pattern is matched here rather than handed to a shell, and it shows you what it would hit before it hits anything.

Recorded automatically whenever a reading crosses the CPU or memory threshold — so an overnight spike still has an answer in the morning. The collector and the ps it runs are left out: a record of a busy moment listing the thing that looks at processes is noise by construction.

The percentage is of what this account may use, not of the machine. Inside CageFS /proc/stat is virtualised to the account's own CPU allowance, so one PHP request working hard can cross 70% while the server as a whole is idle. Raise processes.record_when_cpu_above if these are too frequent to be interesting; Whole machine is where the server's own load lives.

Every other page here measures something being used up — disk, memory, CPU — and none of them can say why. A machine pinned at 100% because it is answering four thousand DNS queries a second looks exactly like a machine pinned at 100% for any other reason, and the difference is only visible on the wire.

What is listening, and what usually answers on each port. A port number is a convention rather than a fact — anything may bind 443 — so the service column is the expectation and the process column is the reality. Where they disagree, that disagreement is the interesting thing.

Who is connected

Established connections grouped by the address at the other end. Grouped because the useful shape is “one address is holding nine hundred sockets”, which a flat list of nine hundred rows states without showing.

The tool of last resort, and treated as one: it opens a live interface as root, runs for as long as you ask, and is the only thing this collector does that could itself be felt on a busy machine. It records counts and endpoints only — how many packets, between which addresses, on which ports, and for DNS the names asked for. No packet contents are stored, ever.

Run when you open this page rather than read from a record — “was it working five minutes ago” is never the question. A nameserver that is busy and answering is a completely different situation from one that is busy and timing out, and only asking it can tell them apart.

Every capture that has been taken. One report answers “what is happening now”; two an hour apart answer “is this getting worse”, which is usually the real question. Rates are shown rather than totals, because two samples of different lengths cannot be compared any other way.

Everything else here watches the machine from the inside, which cannot answer the question a visitor is asking. A server can be healthy by every internal measure and still return 500 because a config file is wrong, or return 200 with an empty page because a database is down — nothing in /proc knows. These are checked from this machine, so they cannot tell you the site is unreachable from somewhere else.

One request, made now, reported in full — what the server actually sends, which is the question while you are setting a check up or arguing with a cache. Type an address with or without https://. The body is shown as text and never inserted as markup; the preview runs in a sandboxed frame of its own, which can neither read this page nor act as you. The body is kept for an hour so the preview can load it, then discarded.

A capture or an access log is a wall of numbers, and the same number means nothing on Monday and something on Friday once somebody has looked it up. This is not your own names — that is what you decided an address means, and is small, hand-written and permanent. This is what the internet says it is: automatic, occasionally wrong, and stale within months.

The lookup leaves this machine, so it is a queue rather than something that happens while you wait. One is resolved per collector run, oldest first — a page showing forty addresses would otherwise make forty DNS queries, and one slow resolver would turn it into a timeout. Anything still waiting can be withdrawn.

A report full of 165.245.160.135 and 8000 is one you decode in your head every time you read it. Give them names once and every network page here shows them. An address can be exact or a whole network in CIDR form, and the more specific one always wins. The number is still shown beside the name — a label is a note about a fact, never a replacement for it.

The same applies to HTTP headers. Ask for a page explains the common ones already, but the interesting headers are usually the house ones a CDN or an application invented and nobody documented. Name one here and its meaning appears beside it, marked as yours.

dentry holding 400 MB and kworker/3:1H at the top of the CPU list are both entirely normal, and both look alarming to somebody who has not met them before. There is a built-in explanation for the common kernel slab caches and processes, shown by the ❓ beside each name on those pages.

What it cannot know is this machine: the in-house daemon, the cron wrapper somebody wrote in 2019, the process named after a project nobody remembers. Write those down here and they appear on the page beside the name, marked as yours, for whoever reads it next — including whoever inherits this server. Yours replaces a built-in one of the same name.

A name ending in * covers a family: kmalloc-* describes every size of it, and php-fpm* every pool.

The whole list, not just the watched ones. Failed and masked are called out because they are the two states that are silently wrong: a failed unit uses no memory and no CPU, so it is absent from every other panel here rather than standing out in one, and a masked one cannot be started at all until somebody unmasks it.

“Needs attention” is chosen by state and never by name — no service here is more important than another, only more broken. It opens on that when anything has failed, because a page that opens on two hundred working units when one has failed has buried the only line that mattered.

A machine misbehaving does not tell you which log to open. This is all of them merged, newest first, because the line that explains it is the most recent one somewhere.

What keeps repeating

A log filling a disk is almost never full of interesting distinct events; it is one event ten thousand times. These are the same lines with their variable parts — numbers, addresses, paths, quoted strings — taken out and counted, which is the only presentation of that fact anybody can act on.

Three months back and twelve forward. Domain expiries and the projected full-disk date are worked out here; renewals, invoices and anything else only you know about are the ones you add. Behind us: the critical alerts that fired, so “what changed around then” has an answer.

Dates you keep

Marked important means it appears on the dashboard and in the alert emails as it approaches, until you mark it dealt with. Marking it dealt with is reversible — it is a claim somebody makes, and sometimes makes wrongly.

A domain expiring is the one outage nothing else here can see coming. Disk, memory, CPU and the HTTP checks all stay green right up to the moment the name stops resolving, and then everything fails at once for a reason that is in no log. Checked weekly — a registration does not change between Tuesdays, and whois servers throttle anybody who asks more often.

A one-off lookup for a name you are not watching — whether it is taken, who holds it, when it expires. Nothing is stored and no domain is added to the list; there is a button for that if the answer makes you want one.

A name resolving is not the same as a zone being right. Mail stops arriving because an MX was dropped or an SPF edited; a subdomain starts pointing somewhere that is not yours; a TTL cut to sixty seconds during a migration is never put back. None of that shows anywhere else here. Names come from a zone transfer where your nameserver allows one, from cPanel's own record of this account's domains, and from what was found last time — not from guessing at names.

What changed

A record that stops resolving is kept and marked gone rather than deleted — “the MX disappeared last Tuesday” is the question somebody asks a week later, and a deleted row cannot answer it.

Two identities that get confused constantly: the browser at the other end, and the account this process runs as. The second is the one that answers permission questions — “this account cannot read that” means nothing until you know which account it is.

How hard the processor is working, and how much of it this account may use. Inside CageFS the account's own percentage is of its allowance rather than of the machine, so a pinned account and an idle server are the ordinary shape of hitting a limit rather than a contradiction.

A processor is never simply "busy". Every jiffy it spends is charged to one of eight states, and which one decides what to do next — work in userland is the machine earning its keep, time in the kernel is a syscall-heavy workload, iowait is a disk problem wearing a CPU costume, and steal is the hypervisor and not you at all. The single busy percentage everywhere else on this dashboard is all of these added up and is the reason the question usually gets asked twice.

The distribution the average has already thrown away. Eight cores at 12% and one core at 100% beside seven idle ones are the same whole-machine reading and completely different problems: the second is a single-threaded bottleneck, it will never make the overall percentage look alarming however stuck it gets, and buying more cores makes it exactly as slow as it was.

Processor use per process, read as root, so this is every account on the machine and not only this one. Percentages here are of one core, the way top reports them — a process with four busy threads shows 400%, which is a fact about it rather than an error. That is the opposite convention to every other percentage on these pages, and the columns say so.

Why a load average of 40 can sit on a machine that is 5% busy. Load is a queue length, not a processor reading: it counts everything runnable and everything stuck in uninterruptible sleep waiting on storage, so a hung disk drives it into the dozens while the cores idle. These are the figures that tell those two apart.

The page for "the CPU is pinned and the server is idle", which is not a contradiction and not a broken reading. On shared hosting this account is given a slice of the machine and told, through a percentage of its own allowance, how much of that slice it is using. Both numbers are correct and they measure different things; nothing useful comes of comparing them as a ratio, so this page puts them side by side instead.

What this account can actually run, and what the machine has that it cannot reach. Those are the same question on ordinary hosting and different ones here — see below.

A hole in the readings is not the same as the machine being off, and the charts distinguish them — red shading where the boot time moved, so it really did restart, and grey where the boot time was unchanged either side and the collector simply was not running. Uptime counts only the first kind; a cron job that stops should not look like a server that keeps falling over.

A dashed red line marks each restart the host knows about from last. Those need not leave a hole at all — a reboot the collector resumes after within a minute has nothing to shade — and they include restarts from before boot times were being recorded, which the readings alone cannot show.

Every deletion, every scan, every request to the host collector, every email it sent, every configuration change and every sign-in — whether each worked or not. A refusal is kept as carefully as a success: knowing what was not done is half of knowing what happened.

Mail goes out through PHP's mail(), which on a cPanel host means the server's own mail queue. One message per alerting run rather than one per rule — a disk filling up trips several rules at once, and five separate emails about the same event is how people learn to ignore them. The message takes the volume of its loudest line, lists them loudest first, links each one to the page that answers it, and ends with the commands to diagnose the same thing from a shell — because the most likely moment for this dashboard to be unreachable is the moment the disk is full, which is exactly when its email arrives.

The two jobs run on different clocks: vitals cost milliseconds, a scan costs minutes of disk work. Only the lines between this dashboard's markers are touched — anything else in your crontab is left exactly as it is.

The host collector

Everything on this site that reaches past the account — memory by user, the machine's disks, whether daemons are alive, the space held by deleted files — comes from a second collector that has to run as a user who can read /proc for every process. The dashboard cannot start it and cannot install it; it can only tell you exactly what to run.

A disk monitor should be able to answer for its own disk use, and should never stop collecting without saying so. Neither is visible anywhere else on this site: a database quietly growing to a gigabyte looks exactly like one that is fine, and a cron job that stopped last Tuesday just makes every chart look quiet.

What this install can and cannot do. The first three are the usual reason a fresh deployment returns errors.

Everything the account's own cron does, and how often. The costs are real and uneven — a reading is milliseconds, a disk walk reads every inode under the account — so each carries a recommendation, and where that recommendation is only when asked it is a decision rather than an oversight. The reason is beside each one.

Everything root's collector does on this machine's behalf. These see what the account cannot — the other tenants, every filesystem, the processes outside the cage — and cost correspondingly more. Anything here can also be asked for on the spot; it happens on the collector's next pass rather than now, and what is waiting is shown.

How many outbound lookups either collector may make per run. These reach registries and public resolvers, which defend themselves: a burst of twenty gets this address rate-limited, and the result is twenty domains reported as unknown — one self-inflicted problem wearing the costume of twenty real ones. Queues drain oldest first, so nothing is starved by newer arrivals.

Every conversation, newest first. Open one to read it, or pick it up where it left off — that starts a fresh conversation carrying what was said, so the limits it ran into do not come with it.

Pinning one keeps it past the retention rule. That rule deletes by age, which is right for the ninety-nine nobody will read again and wrong for the one that explained an outage.

Assistant is a model you configure, given a fixed set of read-only tools for looking at this machine — the name is the job: somebody on watch who tells you what they can see. It is off until you set it up, and when it is on this tool sends parts of your logs, your disk figures and your process list to whichever service you point it at — which is worth deciding deliberately rather than discovering. A local model needs no key and nothing leaves the machine.

The model can never write a query. Every tool is a fixed statement with typed, enumerated parameters bound to it, and anything outside those bounds is refused before it reaches a database. The rest of the settings are on the pages under this one.

Which service to speak to, which model, and what it costs. Prices are per million tokens, in cents, and are optional — without them everything still works and the running total reports token counts rather than money, because a wrong price is worse than no price.

Prices this install has been told about, per provider and model. Changing the model on the Model page fills the price boxes from here rather than making you look them up again — and a model you have used before keeps whatever you set for it. Nothing here is fetched from anywhere: these are the numbers you were quoted, which is the only honest source for them.

What stops a conversation, and how much of one the model can hold at a time. Every limit here is checked before a request rather than after, so it stops the next call rather than being noticed once the money is gone.

Entirely optional, and off until keyed. Generating a picture costs money; searching for an animation is a free read against somebody else's quota. They are limited separately for that reason.

A tool switched off is not described to the model at all, rather than being described and then refused — one told about a tool it cannot use keeps trying it and explains its absence to you as a fault. Anything that asks permission asks every time, and a refusal is reported back to the model so that it works around it rather than retrying.

Runs one tool with parameters you choose and shows exactly what comes back — the same thing the model would receive, in the same shape. Useful for two things: seeing whether a tool actually works on this install before switching it on, and reading what the model is being told when its answer looks wrong. Nothing here involves the model and nothing leaves the machine.

Every conversation, with the tokens it used and what that came to. Costs are shown only when prices are set — without them this reports token counts and no money, because a wrong price is worse than no price.

Almost every it was working yesterday turns out to be something changing underneath — a kernel that came with a package update, an OpenSSL bump that moved a default, a PHP point release that deprecated a function. None of it shows in any other chart here, and the machine does not keep a usable note of it either: the package log belongs to root, is not always present, and records what was installed rather than what is now answering. Every time on this page is when a change was noticed, not when it happened — this runs when capabilities are checked, so a weekly check turns a Tuesday upgrade into a Sunday event.

What changed, by group

Each bar counts the programs whose availability or version moved, coloured by group. Click one to see exactly what. This was on what the host allows, which is a page about what is available now; a history of changes belongs beside the rest of the history.

What each thing is at this moment, and how long it has been that way. Where a program offers no readable version — and plenty do not — a fingerprint of the file stands in: a weaker fact than 3.0.11 → 3.0.13, but the same kind of fact, and the only one available.

This is the present tense of what changed, and the two answer different questions: this one is what you are running, that one is when it started being that. Both are filled in by the same check — what the host allows is where it runs.

Every chart here answers what happened. None of them answer why, and the why is almost never in the data — it is that somebody upgraded PHP on the Tuesday, or that the backup window moved. Six months later the chart still shows the step and nobody remembers the reason. A note is pinned to a moment and drawn on the charts it belongs to.

Forensic copies, written beside the databases and deliberately outside every retention rule — the point of one is to survive the roll-up that would otherwise discard the readings it is about. Which also means nothing expires them, so they are listed here for somebody to look at.

A trusted server is explicitly configured here and checked through HTTPS OpenID Connect discovery. Saving it does not share data or sign anyone in; those authorization-code steps are deliberately separate.

Local package manifests only. Enabling never executes package code. A few fixed, core-owned capabilities may be authorized; requested capabilities remain visible for review.

This page reports Zogograph evidence, not assumptions. It never shows key paths or key material. Change protection choices in the setup wizard; recovery and sealing stay CLI-only so a browser session cannot handle a private key.

Signed packages are reviewed here before any future runtime can use them. Enabling records explicit capability grants and private-package settings; it does not execute package PHP in this Zogograph release.

This dashboard's own log. When a request fails it shows you a reference — find the same reference here for what actually happened. Identical faults are counted together rather than listed each time: the dashboard polls, every poll calls the API, and one broken function becomes forty lines in a minute.

Written on a schedule by Assistant from a fixed set of figures for the period — the same questions every time, so two summaries can be compared. What varies is what it makes of them. Each one is emailed when it is written; this is where they are kept.

This runs what you type, as the user the web server runs as, and shows the output. Be clear-eyed about what that means: anybody who has the dashboard password has this, so it stays off until somebody switches it on here, and every command is written to the action log before it runs. Each command is its own process; the directory it runs in carries over between them and is shown above the box, so cd works and you can see where you are. What does not carry over is anything else a shell remembers — variables, functions, background jobs — because there is no shell sitting between requests to remember them.

It is a transcript, not a screen. Colour, bold, tabs and progress lines that rewrite themselves all render. Text editors such as vi and nano open a constrained browser editor instead. Programs that need an interactive terminal — top, ssh, or anything asking for a password — cannot be driven from here.

Commands that were run close together, grouped the way the work actually happened — an hour's gap starts a new one, because somebody who ran three commands, went to lunch and came back was doing two different things.

Give one a topic and it becomes something you can come back to. A list of timestamps is a list nobody reads; “the disk filling on Tuesday” is a thing you look for six weeks later.

Opening a session shows what was run. What each command printed is behind its own switch, because those are two different questions with two different answers: what you typed is safe on any screen, and what came back may be a config file with a password in it.

A conversation ends and everything worked out in it goes with it. These are the things kept on purpose — which of the installed PHP versions the site actually runs, what the 03:10 spike turns out to be, how you have said you want to be told things. Everything here is editable and deletable by you, and every one records the conversation it came from, because a remembered claim whose origin cannot be found is a rumour.

Kept apart from the account’s own database on purpose: everything here is a record of what was sent to somebody else’s service and what came back, and that is worth being able to read, export or delete without touching the readings. It is also the one database here that grows by conversation rather than by schedule — a long session with a few screenshots in it can outweigh a week of measurements — so what it is holding is worth being able to see.

A monitor that fills the disk it is watching has failed in the most embarrassing way available to it. This is what this account's database holds, what each part of it costs, and — the part that is usually missing — what the file will settle at if nobody touches the settings again. A table set to keep 400 days that has been collecting for thirty is at under a tenth of its eventual size, and nothing else here says so.

Written by root's collector, and read-only from here. This account can usually open it to read and should never be the thing that rewrites it — a rewrite started from here would leave a file the collector could no longer write. Where it cannot be opened at all, that is said rather than shown as zeros.

Zero means keep for ever, and for several of these that is the recommendation rather than an oversight — the note beside each one says what it is protecting. Readings are kept in tiers: full detail for the first stretch, then five-minute buckets, then hourly, then daily, each holding the worst moment in its bucket rather than an average. Nothing within a couple of hours of an alert is ever rolled up or cleared, whatever these say.

The scanner steps over every symlink it meets, deliberately — following one double-counts at best and walks out of the watched directory at worst. The consequence is that a bad link is invisible to this tool while still being the thing that ruins everything else: a link pointing at one of its own ancestors is an infinitely deep directory to anything that does follow links, and backups, du, tar and web crawlers all do. That is how a disk fills up with one file copied ten thousand times.

When a directory of dangling links is the problem, ticking four hundred boxes is not a workflow. Name a directory and everything broken under it goes, however deep — you do not need to know how many levels there are, and you should not have to add a * until the count comes out right. Working links under the same path are left alone and counted: the destructive half is bounded by a fact about each link, not by how carefully the path was typed. You are always shown exactly what would go before anything does.