Automation audit: the rules that fired—and misfired

Automation removed hundreds of small decisions. Nine bad decisions explain why a weekly human review still earns its place.

The audit ran from June 1 through July 30, 2026. We created equivalent merchant renames, category assignments, flags, and recurring-transaction rules in Rocket Money, YNAB, Monarch Money, and Quicken Simplifi where each product supported them. The dataset included repeat merchants, changed descriptors, tips, refunds, transfers, and purchases from stores that span several categories.

How we defined correct

A correct fire produced the intended merchant, category, split, or review flag without hiding information. A miss left a matching transaction unchanged. A misfire changed a transaction that should not have matched or applied the wrong action. We counted outcomes, not just whether a rule was enabled. One overbroad rule could therefore create several errors.

We avoided fragile tests designed only to trap software. The hardest cases were ordinary: a fuel station that also sold lunch, a supermarket purchase containing medicine, a refund using a shortened descriptor, and credit-card payments arriving with reversed sign conventions. Those are exactly where automatic confidence can become expensive.

Days 1–7: broad rules felt brilliant

The first week rewarded ambition. Rules cleaned merchant names, moved predictable bills, and flagged reimbursements before the weekly review. Rocket Money removed the most touches because it paired merchant cleanup with categorization across a broad imported feed. The median review fell from 33 minutes without rules to 14 minutes with them.

Then a café inside a bookstore exposed the flaw. A rule matching the parent merchant categorized a $46 book purchase as dining. The transaction did not look suspicious because the renamed merchant and category agreed with each other. Only the receipt revealed the error. Broad rules are efficient precisely because they can be confidently wrong.

Days 8–30: transfers caused the serious errors

Three of nine misfires involved transfers or payments. Simplifi briefly treated a reversed credit-card payment as income after a descriptor rule ran. Monarch categorized one savings transfer as an expense because the receiving side arrived a day later. Both were correctable, but either could distort “safe to spend” more than a mislabeled lunch.

YNAB’s deliberate approval loop limited harm. Its single misfire assigned a pharmacy purchase to groceries, and the imported transaction still waited for approval. This increased review time relative to invisible automation but made the system easier to audit. The result supports our YNAB review: friction can be a control, not merely a usability cost.

Days 31–60: narrow rules won

We rewrote five broad merchant rules after day 30, matching more specific descriptors and using review flags for ambiguous stores. Correctness improved from 97.7% in the first half to 99.1% in the second. The weekly time saving fell by less than two minutes. Precision did not require returning to full manual work.

Recurring rules performed best when amount and cadence were stable. Rent, insurance, and subscriptions rarely needed intervention. Fuel, warehouse stores, online marketplaces, and person-to-person payments deserved flags rather than automatic categories. A useful system distinguishes boring repetition from merely repeated merchant names.

The maintenance nobody advertises

Rules decay. Merchants change payment processors, households switch categories, and an annual bill may arrive under a new descriptor. We spent 28 minutes across 60 days editing or retiring rules, excluding ordinary transaction review. That is small beside the saved time, but it prevents automation from being “set and forget.”

Our recommended routine is ten minutes weekly and a deeper rule review every quarter. Sort for uncategorized transfers, unusually large auto-categorized purchases, new merchant names, and rules that have not fired in 90 days. Export before a major cleanup when the app permits it.

Which app handled rules best?

YNAB was most accurate because approval made outcomes visible. Rocket Money delivered the largest time saving and the most flexible low-friction routine, so it remains our practical automation winner despite four misfires. Monarch offered strong household visibility, while Simplifi’s watchlists were better for monitoring ambiguity than forcing every purchase into a rule.

These results fit the 90-day Q3 benchmark, where Rocket Money won overall and YNAB won disciplined budgeting. The decision is not automation versus attention. It is choosing which decisions software may repeat and reserving a short, consistent review for everything that can change the plan.

Frequently asked questions