If I have to process the same kind of data repeatedly, check the same condition across hundreds of records, or perform the same sequence of steps every day, I will usually look for a way to turn it into code.
That is one reason I use AI heavily in my work too. Between scripts, APIs, models, scheduled jobs, and coding agents, we no longer need to do a lot of work one item at a time.
But while building these systems, I have noticed that some of the most useful automated checks I have today only exist because I first found the problem manually.
I recently worked on a pipeline that takes unstructured documents, extracts structured data, validates the result, and writes it to a database. The first version was pretty much about getting the document, extracting the fields, validating the response, and saving it.
The pipeline is much more defensive now. Not because I sat down at the beginning and predicted every way it could fail. Most of the checks were added after seeing a real case where the simpler version gave me the wrong answer.
That changed how I think about automation.
A successful run is a very low bar
The first thing I learned was how easy it is for an automated process to succeed technically while producing something useless.
You can get a successful API response, valid JSON, all the required fields, and a successful database insert, and still end up with the wrong result.
I saw an early version of an extraction flow return perfectly valid JSON using fields different from the ones the application expected. There was no failed request and no malformed response. Looking at the raw output manually was what exposed the problem before it became part of the rest of the pipeline.
That sounds obvious once you know it happened, but this is exactly the kind of thing automation makes easy to overlook.
That distinction becomes even more important when AI is involved because model output can look completely reasonable while being wrong in a way your normal validation does not catch.
A number can have the right type and still be the wrong number. Two records can look identical from their metadata and still represent different things. A source document can itself contain a mistake that the model extracts faithfully.
The output doesn't always contain enough information to tell you something went wrong.
Sometimes you have to look at the source.
Manual checks often become code later
I don't think manual checking scales, and I wouldn't want to inspect every result forever.
What I have found useful is treating manual checks as a way of discovering what the automated system should eventually know.
The pattern usually looks like this:
Suppose you discover that a summary value coming from a source cannot always be trusted, but the underlying items used to calculate it are available.
The permanent fix might be something very boring:
function calculateTotal(items) {
return items.reduce(
(total, item) => total + item.quantity * item.price,
0
);
}
In the system I was working on, manual verification exposed cases where values needed to be recomputed from their underlying entries rather than accepted directly from the source. That eventually became an always-on deterministic check.
This is how I now prefer automation to improve over time. A person should not keep solving the same known problem. Once we understand a failure well enough, we should encode that knowledge and move human attention elsewhere.
A warning and a decision are different things
One mistake that is easy to make is giving an automated check too much authority too early.
Imagine you build a rule for identifying duplicate records.
The first version might look like:
if (looksLikeDuplicate(recordA, recordB)) {
delete(recordB);
}
That assumes the heuristic is good enough to make a destructive decision.
A safer first version is:
if (looksLikeDuplicate(recordA, recordB)) {
flagForReview(recordA, recordB);
}
I had an automated audit flag a group of records as likely duplicates based on reasonable metadata similarities. Once the underlying files were compared directly, nine out of ten flagged pairs were actually separate legitimate records. Only one was a real duplicate.
If the heuristic had been wired straight into a cleanup job, the automation would have damaged more good data than it fixed.
I now think about automated checks in roughly three levels: signal, confidence, and action.
A weak signal can trigger review, while a stronger, well-understood rule might trigger an automatic correction.
You only learn how much confidence a rule deserves after seeing where it produces false positives and false negatives.
Manual review is useful during that stage because it gives you the evidence needed to tighten the rule.
In my case, broader duplicate detection eventually became several narrower checks based on the verified failure patterns.
The automation got stricter because the manual investigation showed where the first assumption was too broad.
The system should be allowed to say "I don't know"
As the number of checks grew, manually reviewing everything became its own problem.
The answer was more automation, but with layers.
The flow eventually looked closer to this:
I like this model because it does not force automation to pretend it knows everything. This way the system handles the cases it understands and spends more compute or performs deeper checks on the unusual ones.
And after that, there is still room for uncertainty. In code, the decision can be as simple as:
if (validationErrors.length === 0) {
accept(result);
} else if (canVerifyDeterministically(result)) {
verifyAndAccept(result);
} else {
sendToReview(result);
}
The important decision is allowing sendToReview() to exist.
Once every branch is required to produce an automatic answer, the system starts converting uncertainty into bad decisions.
I would rather have twenty records waiting for someone to inspect than twenty confident answers the system invented because there was no other state available.
Manual review should get smaller as the system gets better
I do not want somebody opening every document, checking every number, or reviewing every record forever.
At the beginning of a new system, though, I think manual inspection is valuable because you are still learning what the real failure modes look like.
You check a few results, find a pattern, and automate it so you no longer spend human time on the same problem. When another strange case appears, you investigate it, understand what went wrong, and turn that lesson into another rule. Over time, the review queue gets smaller, and the cases left for humans become the ones that actually need judgment.
That is a much better outcome than removing manual review on day one and assuming that successful runs mean correct results.
