How ConfigCheckup works out its numbers

The posture score

Every check carries a weight. Most take the weight of their severity: critical 10, high 6, medium 3, low 1 and informational 0; a few checks have their own. A check that passes earns its full weight, a warning earns half, and a failure earns nothing. The score is the points earned out of the points available, scaled to 100.

Checks that could not run, because a data source was refused, failed or needs a licence the client does not have, are left out of both sides. So are checks that do not apply, checks whose risk was accepted, and your own custom checks. A client is never marked down for something ConfigCheckup could not see, but a score from a narrow read is not a clean bill of health: the line beside the score says how many checks it rests on.

When a fix runs, or someone marks a finding resolved, the settings it touched are read again and the score is updated within seconds. The next full assessment confirms it.

Grades and colours

The grade puts the score into a band, so a client can see at a glance where they stand: A from 93, A- from 85, B+ from 78, B from 70, C+ from 62, C from 54, D from 45 and E below 45. It comes from the same weighted score, so it moves when the score moves, including within seconds of a fix.

The letter is the only band. Its colour and its word follow it everywhere, on the dial, in lists and in reports: green for A and A- (Strong), amber for B+ and B (Good), orange for C+ and C (Fair) and red for D and E (Weak). A client that has never been assessed has no grade and is shown in grey.

Like the score, the grade covers only the checks that could be run. A client that could not be read in full can earn a good grade from a narrow set of checks; the coverage line beside the score shows how many checks it rests on.

Scores by area

Each area, such as identity or email, is scored the same way as the whole client: every check in the area earns its weight when it passes, half when it warns and nothing when it fails, and the area’s score is the points earned out of the points available. Checks that could not be run, that do not apply, or whose risk was accepted are left out.

Because areas hold different numbers of checks, and some checks weigh more than others, the overall score is not the average of the areas: an area with several critical checks counts for more.

The number failing counts the checks in the area that fail or warn. Select an area to see them in the findings list. Benchmarks compare your clients on these same area scores.

Severity

How much harm the gap could do. It sets the order to fix things, the weight in the score and the time to fix it. The levels, approved by the owner, are:

Severity also sets a check’s weight in the score (10, 6, 3, 1 and 0, in that order), so fixing one critical finding moves the score more than fixing several low ones. Some checks raise or lower their own severity with what they find, such as more admins without MFA. Within a severity, findings are listed by weight, then by how many users, devices or settings they affect. Service levels measure fixes against these targets.

Checks that could not run

A failed check means ConfigCheckup read the setting and it falls short. That is a finding about the client, and it counts against the score.

A check that could not run means ConfigCheckup could not read what the check needs: Microsoft refused the request, a data source failed, the client lacks the licence, or the service is not set up. Nothing is known either way, so the check is left out of the score and never shown as a pass. Frameworks, Cyber Essentials and baselines show it as not assessed.

Some gaps are expected: a client without Entra ID P2 has no risky-user data, so there is nothing to fix. Others need action, such as a permission the client has not consented to, or an error from Microsoft. Those are listed under Not fully collected, with Microsoft’s own words.

An empty answer is not a gap. A client with no Conditional Access policies has been read, and fails the check.

Licence gaps

Some Microsoft 365 data only exists when the client has the licence that switches it on. Risky users and PIM need Entra ID P2, Conditional Access needs Entra ID P1, and device compliance needs Intune. Without the licence there is nothing to read, so the checks that depend on it are not assessed and are left out of the score. That is not a fault, and there is nothing to grant.

ConfigCheckup checks the client’s licences before it decides. If the client does hold the licence and Microsoft still refuses, it is treated as a problem to look into, not a licence gap, and appears under Not fully collected.

If the client buys the licence later, run Test data sources from Settings to confirm the data can be read, then run an assessment to bring those checks into the score.

Data sources

A data source is one part of Microsoft 365 that ConfigCheckup reads, such as users, Conditional Access, Intune devices or inbox rules. Each check depends on one or more of them.

Read: everything came back. Partly read: some of it did, and the note says what is missing.

Needs a licence, Not set up and Not in this cloud: the client does not have the feature, so there is nothing to read and nothing to fix. The checks that depend on it are not assessed.

Consent needed: Microsoft refused because the client has not granted a permission, and their Global Administrator accepts the read-only link again. Could not be read: Microsoft returned an error. It is often temporary, and the message gives Microsoft’s words.

Test data sources reads every source from the live tenant without scoring anything, so a fix to permissions can be checked straight away. Run an assessment afterwards to bring the newly read sources into the score.

Accepted risk

Accepting a risk records that the business has decided to live with this gap, for a reason, usually until a review date. It is for a gap that cannot reasonably be closed, not one that has not been closed yet.

The finding leaves the open findings straight away, and the score from the next assessment or check. The reason is printed in the report’s appendix. It is not hidden: frameworks still show the control as not met, because it is not. Cyber Essentials and baselines judge it on what the client’s tenant shows.

Only an Admin or Owner can accept a risk. A Consultant can ask; the finding stays open, and in the score, until an Admin or Owner decides. When the review date passes, the check counts again.

Confirming fixes

A fix is only counted when ConfigCheckup reads the tenant again and the check passes. Nobody’s say-so counts, including ConfigCheckup’s own.

Straight after a change runs, ConfigCheckup reads again just what it touched and re-scores the client, usually within seconds. Microsoft can take a while to apply some changes, so if it does not show yet, ConfigCheckup checks again after waits of 10 seconds, 30 seconds, 90 seconds, 5 minutes, 10 minutes, 30 minutes and 60 minutes: about 2 hours in all. Secure Score items and Purview changes, which Microsoft re-scores slowly, are checked over 48 hours.

Some changes cannot show in what ConfigCheckup reads, such as signing someone out; they are marked Applied. A Conditional Access policy created in report-only mode enforces nothing, so its finding stays open until someone switches the policy on. A check that stops applying, rather than passing, says No longer applies.

If the check still fails when the schedule ends, you are told, with troubleshooting steps.

Marked done, still failing

Marking a finding resolved does not close it on its own. ConfigCheckup reads the setting again straight away, and again on the same schedule as an automated fix, to see whether the client now passes.

If it passes, the finding closes and the score goes up. If the schedule runs out and the client still fails, the finding is reopened, keeps counting in the score, and is listed in the report as marked done, still failing, so a report never claims work that did not land.

The usual causes: the change was made in the wrong place (a different policy, or a policy left in report-only), it does not cover everyone the check looks at, or Microsoft has not applied it yet. The troubleshooting steps list what to look at for this check.

Drift

Between assessments, ConfigCheckup reads again the settings behind the findings, to catch changes made directly in Microsoft 365: Conditional Access, authentication policies, admin roles, sharing and device compliance every hour, and users, MFA registration and mail forwarding once a day (every 6 hours where an automatic fix policy covers the client).

When something has changed, in either direction, the affected checks run again and the findings and score are updated in place, within minutes. A check that finds nothing changed records nothing.

If one of the critical controls stops passing (MFA, legacy authentication, the identity baseline, admin accounts or external forwarding), admins are alerted, in Teams or Slack too where connected, and the client’s Overview shows it under Monitoring. The finding itself is updated, so it is fixed from there like any other. Where an automatic fix policy covers that check, the fix runs.

Drift checks come with Monitor and above, while drift alerts are switched on in Settings.

Frameworks, Cyber Essentials and insurance

Each framework control is linked to the checks that show whether the client meets it. A control is met when all its checks pass, not met when all fail, and partly met otherwise, including when a check only warns. The percentage counts met controls in full and partly met ones as half, out of the controls that could be judged.

Two kinds of control are left out of the percentage. Not assessed: its checks could not run this time. Your own records: Microsoft 365 cannot show it (a written policy, staff training, physical security), so you evidence it yourself.

An accepted risk still counts as not met here: the control is not in place, and the business has chosen to live with that.

Readiness shows how close the client is on the parts Microsoft 365 can evidence. It is not a certification or an auditor’s opinion.

Cyber Essentials

Cyber Essentials is the UK government-backed scheme, certified by IASME. ConfigCheckup evidences the parts that live in Microsoft 365: accounts, MFA, admin rights, device configuration and supported systems. The rest, such as boundary firewalls, routers and anti-malware on every device, cannot be seen from the cloud, so you record them as attestations with a note.

It is judged more strictly than other frameworks, because the scheme requires each control to exist. A warning counts as not met. A check that could not run leaves the requirement unproven, not met. An attestation can settle a requirement the client’s tenant cannot evidence, but never overrides a failing result from the tenant itself.

Ready means every requirement is met or attested. Needs evidence means nothing fails but something is unproven. Gaps means at least one requirement is not met.

Insurance answers

Each question insurers commonly ask is linked to the checks that answer it. Yes: every linked check passes. No: every linked check fails. Partly: some pass and some fail, or any only warns. Checks that could not run are left out; if none could run, the question is left for you to answer.

Some questions cannot be answered from Microsoft 365 at all, such as backups, EDR on every device, staff training and an incident plan. They always need your answer. Any checks shown beside them are related evidence only.

Your own answer always replaces the automatic one, and is shown with your name.

Wording follows common UK and US proposal forms without copying any one insurer’s. Where an insurer’s form asks differently, answer their form.

Service levels and benchmarks

Each finding has a target by severity, counted from the day it was first seen. A fix counts on the day a later assessment or check sees the client pass, so the times are what you can show a client, not when someone said it was done.

Within target is the share of fixes that landed inside their target. Median time to fix is the middle value, so one very old finding does not distort it. Slipped back counts controls that were fixed and then failed again. Open and overdue counts findings still open past their target, leaving out accepted risks.

Fixes are credited to whoever marked the finding resolved, or else its owner. A due date set on a finding is a reminder for your team; it does not change the target used here.

Benchmarks place each client among your other clients, by score and by area, once you have at least 3 clients assessed. A client only ever sees its own position.

Security events and alerts

Every 15 minutes (every 5 minutes on Automate and above), ConfigCheckup reads what has changed in each client’s Entra ID audit log and sign-in log, and their Microsoft Defender alerts. It raises an event for the steps attackers commonly take: an admin role given, a risky app consent, an MFA method removed, Conditional Access switched off, sign-ins from places too far apart to travel between, many wrong passwords, MFA prompts declined, and more.

Each detection is deliberately narrow. A new country only counts once the client has 14 days of history, and only if nobody has signed in from it for 90 days. Password spraying needs one address failing against 10 or more accounts, password guessing 15 wrong passwords for one account, and MFA fatigue 3 declined prompts; your workspace can change these. High means it may already have worked, such as a successful sign-in after guessing; medium means look soon.

Only sign-in and audit details are read: who, when, from where and what changed, never mail, files or chats. Security events come with Monitor and above.

The thresholds each workspace can change, with their defaults and the range allowed:

Microsoft Defender alerts

Where the client has Defender licences, Defender sees what the Entra logs cannot: malware on a device, a phishing link clicked, a suspicious inbox rule. ConfigCheckup reads Defender’s alerts with the read-only app and raises each medium or high one once, as a security event.

Some are left out on purpose. Low and informational alerts stay in Defender. Data loss prevention and insider risk alerts are about what mail and files contain, which ConfigCheckup never reads. Entra ID Protection alerts would arrive twice, because high-risk users are already read directly. Alerts closed in Defender as a false positive or expected activity are not raised.

Only Defender’s own description is kept: the alert’s title, category and severity, and the names of the accounts, devices and apps it names. If Defender names one account, you can contain it from here.

If Defender cannot be read, this says so, so no alerts shown is never mistaken for none raised.

Your own alert rules

An alert rule names what an Entra audit log record must say: the activity or the category (a rule needs at least one, so none can match every change), whether it succeeded, who made it and what it changed. Any record that matches becomes a security event under your rule’s name, at the severity you choose, for the clients with the tags you pick.

Rules run on the records each security event check has already read, so they cost no extra requests to Microsoft and are checked as often as the built-in alerts.

To stop a broad rule flooding the alerts, a workspace can have up to 25 rules, and one rule raises at most 10 events at one client in an hour; further matches that hour are not raised. Changes made by ConfigCheckup’s own apps can be left out.

The built-in detections’ thresholds can be raised or lowered here, within limits. Your own rules come with Automate and above.

Containment

Stops anyone else using an account. Automatic only for near-certain takeovers, under an approval you give in advance. How it works, and how to set up automatic containment, is in containing accounts automatically.

Licence savings, billing and Copilot

Savings are worked out from the latest assessment, with no extra permissions, in three ways.

Larger plan than needed: people on a top plan where none of the features only that plan adds shows any sign of use. The saving is the difference in price. Accounts already counted as unused are left out, so nobody is counted twice.

Unused seats: paid licences on disabled accounts, and on accounts nobody has signed in to for 90 days. The saving is the full price of each.

Copilot: seats nobody has used for 30 days, leaving out seats given out in the last 30 days, and seats bought but never given to anyone.

Monthly figures are shown as a year’s worth. Prices are yours where you have set them, otherwise Microsoft’s UK list price less your discount, excluding VAT, in pounds. Licences with no known price are left out of the total. Some things cannot be seen from Microsoft 365, such as Teams Phone use or an annual commitment, so each suggestion lists what to check first.

Billing check

The billing check compares what your distributor or Partner Center invoice says you pay for with what each client’s tenant actually holds.

Billed but not held: the invoice has more seats than the client, often a reduction or cancellation that did not reach the invoice. Not assigned: seats billed and held, but given to nobody. Both count as money at stake. Not on this invoice: the client has seats the invoice does not cover, bought elsewhere or not billed on; worth checking, but not counted as waste. Trial: it will end or convert on its date.

Money at stake uses the invoice’s own price per seat where it gives one, then your prices, then list price.

It compares seats and assignments, not whether people use them; that is what Licence savings looks at. Lines that cannot be matched to a client or licence are listed so you can map them by hand.

Copilot usage

Microsoft’s Copilot usage report records the last day each person used Copilot, and in which apps. ConfigCheckup reads it with the assessment and sorts each seat. In use: used in the last 30 days. New: given out in the last 30 days and not used yet, so a roll-out in progress is not counted as waste. Not using it: everyone else.

The share in use leaves out seats given out in the last 30 days, used or not, so a roll-out neither raises nor lowers it. Not used in the report period means Microsoft’s report shows no use in its window.

Not assigned is seats bought and given to nobody. If the client hides user names in its reports, usage cannot be matched to people, so unused seats cannot be listed; turning that setting off in the Microsoft 365 admin centre fixes it from the next assessment.

Unused and spare seats are costed in Licence savings, where they can be reclaimed.

Device lifecycle budgets

Each device from Intune and Entra is checked against two clocks. Hardware age: worked out from the model’s generation where the name gives it, otherwise from when it was enrolled, shown as a minimum. OS support: whether its operating system still gets security updates, and whether the hardware can move to one that does.

Replace now: past its refresh cycle, or stuck on an unsupported system. Plan replacement: reaches either within your planning window, 12 months unless your workspace sets another between 30 days and 730 days. Update OS: the hardware is fine but the system needs updating. Review: its age, or whether it can take a supported system, could not be told.

Each budget level suggests a replacement per device type, with a minimum specification and an indicative UK price, excluding VAT. The cost is the number of devices times that price, so it is for budgeting, not a quote. Virtual machines and Cloud PCs are never costed. The forecast puts each device in the year it falls due, and anything overdue counts this year.

Settings backups

ConfigCheckup keeps dated copies of the tenant settings that are hardest to rebuild by hand: Conditional Access policies, named locations, SharePoint and OneDrive sharing, user and guest permissions, authentication methods, security defaults, Intune compliance policies, Intune configuration profiles (templates) and cross-tenant access defaults. A copy is taken after every assessment, and once a day on Monitor and above. A new version is stored only when a setting changed, and the last 30 versions of each are kept.

Conditional Access policies, named locations, SharePoint and OneDrive sharing, user and guest permissions, security defaults, Intune compliance policies, Intune configuration profiles (templates) and cross-tenant access defaults can be put back from here, as a change with a dry run, approval and a way back. A policy that still exists is updated; one that was deleted is created again, with a new id. Restoring Conditional Access or named locations always needs two approvers, and a restored policy is enforced at once in the state it was saved in, so check it will not lock anyone out. The other settings show what to set back by hand.

This is not a backup of mail, files or chats. Those need a separate backup service.

Plans and the read-only workspace

Every plan shows every finding, with its evidence and step-by-step fix guide. What changes between plans is what ConfigCheckup does for you.

Monitor watches: scheduled assessments, drift and account takeover alerts, reports and a live client portal. It makes no changes in a client’s tenant; you follow the fix steps yourself.

Automate makes changes: fixes and everyday admin with a dry run, approval where needed and a way back, containment in one step, fixes that run under a policy you approve, custom checks, webhooks and the write API. Scale is Automate for more clients, with longer history.

Each plan includes a number of clients, and more can be added one at a time for a monthly price, without changing plan. The trial includes everything in Automate for 30 days, on up to 3 clients.

The read-only workspace

A workspace with no plan, because the trial or subscription ended, or whose payment did not go through, becomes read-only. Nothing is deleted: every client, assessment, finding, report and audit entry stays here to view, export and download.

What stops: new assessments and reports, adding clients, scheduled assessments, drift checks and security alerts, and every change, including automatic fixes and containment. With no plan at all, client portal links stop working too, and clients see that the link is no longer active.

When a renewal payment fails, the workspace keeps working for 7 days while it is put right, and a banner shows the date. After that it becomes read-only.

Choosing a plan, or sorting the payment, brings everything back as it was, including schedules.

Prices and everything each plan includes are on the pricing page.