Conditional Access policies left in report-only mode

What this check looks at

Identifies Conditional Access policies that are not enforcing, which usually means an abandoned pilot.

A Conditional Access policy is in one of three states: On, which enforces it; Report-only, which evaluates it at every sign-in and records what it would have done without doing it; and Off. Report-only is the right way to start a policy, but it is meant to last a week or two while the sign-in logs are checked, not indefinitely.

The check lists every policy that is not On. It warns when one or two policies are in report-only or off, and fails when three or more are, giving the share of the policy set that does nothing. It does not judge whether those policies would be good to enforce; it reports that they are not enforcing.

Abandoned pilots cause two problems. They give a false sense of control: a policy named "Require MFA for all users" sits in the list looking like protection while enforcing nothing. And they make the policy set hard to reason about: the next engineer cannot tell which policies matter, and is reluctant to touch any of them.

Why it matters

Report-only policies give the appearance of control during an audit while enforcing nothing. They also accumulate: the next engineer cannot tell which policies are load-bearing.

A common pattern at MSP onboarding: the previous provider built a sensible baseline in report-only mode, never switched it on, and recorded MFA as done. The client answers yes to "Is MFA enforced?" on a cyber insurance proposal; months later a mailbox is compromised and the insurer finds the policy never enforced anything. The claim and the client relationship both suffer.

Disabled policies carry their own risk. They often hold exclusions or settings nobody remembers the reason for, and switching one on years later, without checking its impact, can lock people out of the apps they need on a Monday morning. Report-only policies also add an entry to every sign-in log record, which is noise during an investigation.

Access control standards such as ISO/IEC 27001 control A.5.15 (access control), which this rule maps to, expect access rules to be defined, applied and reviewed rather than drafted and left. The CIS Microsoft 365 Foundations Benchmark and the CISA SCuBA baseline test whether the required policies are enforced, not whether they exist.

How to fix it

Where: Entra admin centre → Protection → Conditional Access

Before you start

• For each policy in report-only, open its report-only results in the sign-in logs and check what it would have blocked.
• Delete policies that were experiments or duplicates rather than switching them on.

Step by step

1. 1Back up the current policies. Before changing anything, export the policies. In Microsoft Graph PowerShell run Connect-MgGraph -Scopes Policy.Read.All, then Get-MgIdentityConditionalAccessPolicy -All | ConvertTo-Json -Depth 10 | Out-File ca-policies-backup.json, and keep the file with the client’s records.
2. 2List the policies that are not enforcing. In the Microsoft Entra admin centre, go to Entra ID > Conditional Access > Policies. Sort or filter by State and note every policy showing Report-only or Off, with its last modified date. The evidence on this finding lists the same policies.
3. 3Check what each report-only policy would do. For each report-only policy, go to Entra ID > Monitoring & health > Sign-in logs, open a sample of recent sign-ins and look at that policy’s result on the Report-only tab, or use the Conditional Access insights and reporting workbook if a Log Analytics workspace is connected. Note which users or apps it would block or prompt.
4. 4Decide each policy’s fate with the client. Switch on a policy whose effect is understood and wanted. Delete one that duplicates an enforcing policy, was a test, or has been superseded. For an Off policy, find out why it was turned off and delete it unless there is a documented reason to keep it ready. Agree the list with the client before changing anything users will notice.
5. 5Switch on the policies you are keeping. Open each, check the emergency access accounts are excluded, set Enable policy to On and select Save. Switch on blocking or device-compliance policies one at a time, a day apart, so any problem is easy to trace back to its policy.
6. 6Name what remains clearly. Give each policy a name that records its number, who it targets and what it does, for example "CA03 – All users – Block legacy authentication". A policy deliberately left Off should say why in its name, so the next engineer does not switch it on by mistake.
7. 7Confirm. Refresh the policy list: every policy should be On, apart from at most one or two deliberately left Off with the reason in the name. Re-run the assessment; it passes when every policy is enabled and warns with one or two that are not.

What users will notice

Only when a report-only policy is switched on, and then only as that policy describes.

How to confirm it worked

Every remaining policy is On, or intentionally Off with a note in its name explaining why.

Frameworks and controls

This check maps to controls in 14 frameworks. Passing it is evidence that the setting is in place, not a certification against any of them.

The framework mappings were last checked against the published standards in October 2026.

Microsoft’s documentation

• Conditional Access report-only mode (Microsoft Learn)

Related checks

• How to require MFA for every user with Conditional Access
• Security defaults vs Conditional Access: which to use?
• How to block device code sign-in with Conditional Access
• How to set up the user risk policy in Conditional Access

The full Microsoft 365 security checklist, worst first