Work out whether you need a model or just a workflow

The difference in one sentence, and why it matters commercially

Automation executes a rule that a human has already worked out. When this form is submitted, create this record, notify that person, and if the value exceeds a threshold, route it for approval. The behaviour is deterministic. Given the same input it produces the same output every time, it can be tested exhaustively, and when it breaks it breaks loudly in a way an engineer can trace.

A language model estimates an answer to a question nobody could write a rule for. Read this supplier email and tell me which purchase order it relates to. Summarise this call and identify whether the customer is at risk of leaving. Extract the line items from this scanned invoice, which is slightly rotated and from a supplier you have never seen. The behaviour is probabilistic. The same input can produce slightly different output, correctness is a distribution rather than a guarantee, and failures are quiet.

Three questions that decide it

You can get most of the way with three questions. Is the input structured or unstructured? Is the rule stable? What does a wrong answer cost? Answer them about a specific task rather than about a department, because most real processes are a chain of tasks and the right answer differs at each link.

Structured input means the data arrives in known fields: a form submission, an API payload, a row in a database, a fixed format file. Unstructured means free text, an email thread, a PDF from a supplier who changes their layout, a phone call, a photograph. A stable rule is one a competent person could write down completely, and where the written version does not need editing every fortnight. Cost of error covers money, regulatory exposure and trust, and it should be assessed at the volume you actually run, because a one per cent error rate on twenty items a month is very different from one per cent on twenty thousand.

Input	Rule	Cost of a wrong answer	Build this
Structured	Stable	Any	Automation. A model adds cost and removes certainty
Structured	Shifting or fuzzy	Low to medium	Automation with a model for the judgement step only
Structured	Shifting or fuzzy	High	Rules plus a model proposal, with human approval before action
Unstructured	Stable once extracted	Low to medium	Model for extraction, then deterministic processing
Unstructured	Stable once extracted	High	Model extraction with confidence thresholds and a review queue
Unstructured	Shifting	Low	Model, monitored, with an easy correction path
Unstructured	Shifting	High	Model as an assistant to a person. Do not automate the decision

What plain automation still does better

Most of the value in a typical Australian business sits in work that never needed a model. Data is copied between systems by hand. Somebody rekeys an order from an email into an ERP. A spreadsheet is emailed around and merged on a Friday. Approvals sit in an inbox until someone remembers. Reports are assembled manually every Monday. These are integration and workflow problems, they are solvable with deterministic tooling, and they typically pay back faster than anything involving a model.

Practically this means tools like n8n, Zapier, Power Automate or purpose built API integrations handle a large share of what gets briefed as an AI project. If the brief is really about moving structured data between systems reliably, that is workflow automation, and calling it AI just makes it more expensive to build and harder to explain to an auditor.

• System-to-system data movement where both ends have an API
• Scheduled reporting, reconciliation and exception reports
• Record creation and status synchronisation between CRM, accounting and operations systems

Where a model genuinely earns its place

There is a set of problems where no rule will ever be sufficient, and for those a model is not hype, it is the only practical option. The common thread is unstructured input arriving in a form that varies without limit, where writing rules means writing a new rule for every variation forever.

Document processing is the clearest example. Supplier invoices, delivery dockets, purchase orders, remittance advices and certificates arrive as PDFs and scans in hundreds of layouts. Template based extraction works until a supplier changes their letterhead. A model reads the document the way a person would and generalises across layouts it has never seen, which is exactly the capability that rules lack. That is the case for document AI rather than another parsing script.

• Extraction from documents whose layout you do not control
• Classification and routing of free text such as emails, tickets and enquiry forms
• Summarisation of long material where the alternative is a person reading all of it
• Search and question answering over your own documents, grounded in retrieved sources, which is what sits behind most internal assistants

Token cost at volume, and how to work it out before you build

Pilots mislead on cost because they run at pilot volume. A process that costs a trivial amount to run over fifty documents can cost a meaningful monthly figure over fifty thousand, and the surprise usually arrives in month three when the pilot is scaled without anyone redoing the arithmetic. Do the calculation during scoping, at production volume, using published rates for the model you actually intend to use.

The mechanics are straightforward. Models charge separately for tokens sent in and tokens generated out, and a token is roughly three quarters of a word as a working approximation. What people underestimate is everything sent alongside the user's actual question: the system prompt, the few shot examples, the retrieved context, the conversation history and the tool definitions. In a retrieval based system the retrieved context often dwarfs the question itself, and it is paid for on every single call.

Then multiply by the calls per item, which is frequently more than one. An agent that plans, calls a tool, reads the result and responds has made several round trips, and each one resends the accumulated context. Chained or multi step designs can be an order of magnitude more expensive per item than a single call, which is worth knowing before choosing the architecture rather than after.

• The cheapest token is the one you never send. A deterministic filter that removes eighty per cent of items before the model runs is the biggest single saving available
• Model tiering works well: a small model handles the routine majority and escalates only uncertain cases to a larger one

1. Write down the volume: items per month, and calls per item including retries and any multi step reasoning
2. Count the tokens in a real request, not an idealised one: system prompt, examples, retrieved context, history, tool definitions and the input itself
3. Multiply through at published rates for the specific model, and then multiply again by a growth factor for the volume you expect in twelve months
4. Test whether a smaller, cheaper model passes your evaluation set. It very often does for extraction and classification
5. Look for structural savings: caching stable prompt prefixes, tightening retrieval so less context is sent, batching where latency does not matter, and filtering deterministically before the model sees anything
6. Set a monthly budget alert and per key spend limits before go live, not after the first unexpected invoice

Evaluation: build the test set before the system

This is the discipline that separates systems that work from demonstrations that impressed everyone in a meeting. Before building, assemble a labelled set of real examples with the correct answers recorded, drawn from your own data and covering the awkward cases as well as the easy ones. A hundred to two hundred examples is usually enough to be informative and small enough that you will actually maintain it.

Then decide what correct means, in measurable terms, per field or per decision. For extraction, exact match on some fields and tolerant match on others. For classification, accuracy plus the confusion pairs that matter commercially, because confusing two harmless categories is not the same as confusing a refund with a complaint. For generation, a rubric scored consistently, ideally by a person on a sample even if a model does the first pass.

• Use real data, including the messy documents everyone wanted to exclude from the sample
• Track accuracy, cost and latency together, because a change that improves one usually worsens another
• Include adversarial cases: empty inputs, wrong document types, contradictory content, and attempts at prompt injection
• Set a launch threshold in advance and be willing to not ship if it is not met
• Re-run the whole set on every prompt, model or retrieval change, and feed production failures back in so it becomes a regression suite

Human in the loop without creating a second job

Human review is often described as a safety net and then designed as a bottleneck. If a person must check every output, the system has moved work rather than removed it, and staff will route around it within a month. Effective review is selective: the system decides which items a person sees, based on confidence and consequence, and everything else flows through.

Confidence thresholds do most of the work. Set a level above which output is accepted automatically, a band that goes to a review queue, and a floor below which the item is rejected outright and handled the old way. Calibrate those levels against your evaluation set rather than choosing round numbers, then adjust as real data arrives. Consequence matters independently of confidence: a high value invoice or an irreversible action deserves review even when the model is sure.

APP 8 and sending data offshore

Most commercial models are hosted overseas, so the moment you send customer information to one you are dealing with cross border disclosure under the Privacy Act 1988. Australian Privacy Principle 8 requires that, before disclosing personal information to an overseas recipient, you take reasonable steps to ensure that recipient does not breach the Australian Privacy Principles. Section 16C reinforces this by making the disclosing entity accountable for the overseas recipient's handling in many circumstances, which means you cannot outsource the obligation with the data.

There are exceptions in APP 8.2, including where the individual has given informed consent to the disclosure after being told the protections may not apply, and where the recipient is subject to a substantially similar law that the individual can enforce. Consent has to be genuine and specific, which is a higher bar than a line in a privacy policy nobody reads. Separately, APP 1 requires your privacy policy to state whether you are likely to disclose personal information overseas and, where practicable, the countries involved. If your policy does not mention it and you are sending data to a foreign hosted model, the policy is out of date.

The practical design responses are well understood. Strip or tokenise personal information before it leaves your systems, so the model receives the text it needs without the identifiers it does not. Use a deployment in an Australian region where the provider offers one. Get contractual commitments on retention, on whether your data is used for training, and on sub processors. Keep a record of what is sent where, because you cannot assess a breach you cannot describe, and the Notifiable Data Breaches scheme applies to eligible breaches involving these systems exactly as it does to any other. Where the data is health, financial or government related, get privacy advice before the pilot rather than before the launch.

• Prefer providers offering Australian or contractually constrained regional processing for sensitive workloads
• Bring security review into the design phase rather than treating it as a pre-launch sign off

The pattern that usually wins

In practice the systems that survive contact with a real business are hybrids. A deterministic pipeline handles intake, validation, storage, routing and action, and a model is invited into exactly one step where judgement over unstructured input is required. Everything before and after that step is testable, traceable and cheap.

A supplier invoice process is the archetype. The email arrives and is captured deterministically. Attachments are classified, and anything that is not an invoice is filtered out before it costs a token. The model extracts the fields from the ones that remain. The extracted values are then validated against your own data with plain code: does the ABN exist, does the purchase order number match, do the line items total correctly, is the supplier known, does the amount fall within tolerance. Items passing validation post automatically. Items failing go to a review queue with the discrepancy highlighted. The model does the one thing rules cannot, and rules do everything else.