Publication

Who decides? Australia’s new guidance on automated decision-making raises the bar

AI Robot Brain AI Robot Brain

The Office of the Australian Information Commissioner (OAIC) just scraped in and published, on the last day of September, a fact sheet on Australian Privacy Principles (APP) entities’ new obligations under APP 1.7-1.9 (Fact Sheet), along with an accompanying flow chart (Flow Chart). It has also updated the APP Guidelines to incorporate, in detail, new guidance on updated APP 1 (Guidelines).

Top tip – Both sets of information are important reading, but the guidelines are more useful for detailed legal analysis, with the Fact Sheet and Flow Chart serve as a “practical” way for privacy experts to explain the new obligations to their non-expert colleagues. [SPB1.1]

What are the new obligations?

For those who need a refresher (understandable, given the pace of privacy-related change), APP 1.7 will require the following from 10 December 2026.

Where:

  • An APP entity has arranged for a computer program to make, or do a thing that is substantially and directly related to making a decision

  • The decision could reasonably be expected to significantly affect the rights or interests of an individual

  • Personal information about the individual is used in the operation of the computer program to make the decision or do the thing that is substantially and directly related to making the decision

Then the APP entity must include the following information in their privacy policy, as mandated by APP 1.8:

  • The kinds of personal information used in the operation of such computer programs

  • The kinds of such decisions made solely by the operation of such computer programs

  • The kinds of such decisions for which a thing, that is substantially and directly related to making the decision, is done by the operation of such computer programs.

APP 1.9 provides examples of decisions that may be seen as significantly affecting an individual’s rights or interests.

What is a computer program?

Top tip – It’s more than AI. The OAIC has taken a deliberately broad view of “computer program”, noting that it would apply to:
• Pre-programmed rule-based processes
• AI and machine learning processes
• Software, apps or word-processing tools
• Generative AI used to generate text, images, videos, code or synthesis, including chatbots

While broad, this by itself will not extend the application of APP 1. Specifically, falling within the definition of “computer program” does not, by itself, trigger the transparency obligations set out above. The computer program must either make a decision or do something which is “substantially and directly” related to making that decision before these obligations apply. In addition, the decision must meet a stated threshold in terms of its effect: see When does a decision “significantly affect” rights or interests?

Even so, the OAIC's broad interpretation suggests that organisations will need to assess technologies that would not, in our view, obviously fall within the natural meaning of “computer programs”, like Excel spreadsheets with basic formula capabilities. The privacy commissioner recognises that this will be no small task for organisations and will likely require additional or re-directed resource.

When does a computer program do a thing that is substantially and directly related to a decision?

Borrowing from the Explanatory Memorandum (EM) to the to the Privacy and Other Legislation Amendment Bill (Cth), the OAIC suggests that to meet this threshold, a “thing” made or done by a computer program must be both a “key factor” in the decision-making (substantial) and have a “direct connection” with the decision-making (directly related). The EM anticipates that this could be through a computer program recommending or guiding a human decision-maker to a specific outcome.

In the Fact Sheet, the OAIC goes one step further and identifies a computer program as being substantially and directly related to a decision if it:

  • Advises of an appropriate outcome of a process

  • Determines the outcome of a process

  • In any other way influences the outcome of a process (emphasis added)

Equally, the Flow Chart includes as one of its recommended steps assessing whether a computer program is being used to “inform”, or in a way which is “related to”, a decision.

Top tip – This threshold is low. In particular, the Fact Sheet does not set any limit on the level of “influence” required before a computer program will be seen as influencing the outcome of a process. Example 1 in the Fact Sheet is not particularly helpful, as it provides a (very clear) scenario of medical staff using a spreadsheet, which ranks individuals by reference to their medical needs, to decide which individuals to contact and schedule medical appointments for.

When challenged on the span of these terms, the privacy commissioner reiterated the importance of APP entities “going back” to the language of APP 1.7 (specifically, “substantially” and “directly”) to understand their legal obligations. The commissioner noted that not everything that informs a decision would, necessarily, be substantially and directly related to that decision. The commissioner’s response suggests that the guidance issued by the OAIC is deliberately broader than what is required by APP 1.7, resulting in additional work for the regulated community as they sift through all decisions made using computer programs.

Practical example – In the context of candidate screening for a lawyer role – which meets the threshold of a sufficiently “significant” decision according to Example 5 – would use of AI to remove any candidate who does not have a law degree “influence” the outcome of the decision-making process? This is a common use of automated technologies and would not produce a result that is different from that of a human decision-maker (who would have programmed the AI to exclude individuals without a law degree). But it is also true that the use of AI influenced the outcome by actively preventing candidates without a law degree from reaching the next stage.

How much human involvement is enough to avoid the application of APP 1.7?

The Fact Sheet is clear that a decision may be within scope of the transparency obligation even where a computer program output does not replace the entire decision-making process or is subject to human review. For example, in Example 3 in the Fact Sheet, a human is able to override the output of a generative AI program (known as “GPTea” and used to produce profiles of staff for performance and remuneration review), but the output is still always relied on by humans when documenting the reasoning behind a decision, bringing it within scope of APP 1.7.

When considering if an advisory output is within scope of this limb of the APP 1.7-1.9 transparency obligation, the Guidelines recommend that APP entities consider factors including:

  • Which part of the decision-making process the computer program was used for

  • The sources of information used to generate the output

  • The parameters provided to the computer program to generate a decision

  • How the computer program output ultimately influenced the human decision-maker’s judgement

The Guidelines also provide a non-exhaustive list of items to enable an APP entity to assess whether a computer program substantially facilitates and is directly connected to a human decision-maker’s decision.

Top tip – To avoid the application of APP 1.7, the OAIC suggests that humans actively challenge automation bias – i.e. they interrogate the output of computer programs, including by conducting further review of any key performance indicators (KPIs) behind the decision, setting GPTea parameters more narrowly, using additional evidence before making a decision and documenting when they have diverged from GPTea’s recommendations. While these are useful tips, we expect that in practice organisations will find it difficult to distinguish between the varying degrees of human input required by the OAIC in its examples (many of which could be argued either way) and may choose to over-disclose to comply with APP 1.7.

When does a decision “significantly affect” rights or interests?

According to the OAIC, this will take place when the following conditions are met:

  • Significantly means the impact of the decision or thing must be more than trivial and must have the potential to considerably influence the circumstances or outcomes for the individual concerned

  • A right is a moral or legal entitlement to have or be able to do something

  • An interest refers to a concern, benefit, stake or claim relating to something

While it is not referenced in the language of APP 1.9, vulnerability is also relevant, to the extent that a decision may have a greater impact on persons experiencing vulnerability, as opposed to the general population.

The OAIC gives examples of decisions that might fall within the scope of APP 1.9 in Appendix 1 to the Fact Sheet, ranging from those which clearly meet the threshold (like computer programs used to prioritise the provision of health or disability services) to those where the significance of the decision will depend on its context (like programmatic advertising relating to the decision to sell significant goods or services, or differential and/or personalised pricing practices used by online retailers to sell significant goods).

Top tip – The Issues Paper originally published by the OAIC raised the possibility of an ecommerce company selling a book at different price points, depending on the purchaser’s post code, as falling within APP 1.9. Thankfully, the newly-released materials from the OAIC do not go so far – but they are still very broad. In particular, Example 4 in the Fact Sheet makes clear that a decision caught by APP 1.9 would include access to “essential goods”, such as a shopping cart which includes “baby formula, gluten free foods, pharmacy items and over-the-counter medications”, and would also cover decisions relating to programmatic advertising or differential pricing for goods (for example, different pricing for baby car seats depending on a purchaser’s post code).

Practical example – APP entities, especially those who sell groceries or other essential goods (which could include baby products), will need to consider if automated decisions around either the sale or the pricing of such goods need to be disclosed in their privacy policy. Example 4 in the Fact Sheet suggests that even if the pricing differences between individual goods are minimal (the figure given is AUS$0.89), when taken cumulatively, the overall pricing difference may be “significant” as “even a small difference can compound over time to be a significant quantum…”

The Fact Sheet gives the example of gluten-free food as a product that may have a “significant” effect on an individual’s rights and interests. It will be difficult for APP entities to distinguish between goods which are “essential” for individuals with special dietary needs and those which may be purchased based on personal preference (for example, non-dairy milk). Instead of becoming mired in the detail of individual goods or services, we recommend that clients take a wholistic view of the decisions made by computer programs, and their overall impact on individuals.

In addition, while vulnerability may be a relevant factor, many APP entities are unlikely to have this level of visibility. For example, the Guidelines recognise that a decision to suspend a customer’s account with a rideshare company “may have a relatively minor impact on some customers [but] the decision could significantly affect” a vulnerable customer’s rights and interests. In practice, though, a rideshare company will not be able to distinguish between the effect of this computer program on their customer base, and so will need to make the relevant APP 1 disclosures in respect of all decisions made which could affect vulnerable customers, if they intend to keep using the technology.

What does it mean to “arrange for” a decision?

Notably, entities that procure, configure, authorise, integrate or rely on a third-party computer program may still be in scope of the APP 1.7-1.9 transparency obligation. The OAIC is clear that an APP entity has “arranged for” a computer program’s activities even if it does not operate the program itself but is still responsible for deciding to use it. This includes Example 2 in the Fact Sheet, where an APP entity procures off-the-shelf software to generate recommendations for it.

Top tip – APP entities “responsible” for the use of computer programs still need engagement from their third-party suppliers. Specifically, the Guidelines require APP entities to understand how a computer program is being used in a third-party product or service to make or assist decisions, as well as what types of decisions are being made, and to ensure that responsibility for ultimate decision-making is made clear in their contracts.

Practical example – Along with the anticipated introduction of “controllers” and “processors” under the Privacy Act, changes to APP 1.7-1.9 give APP entities a basis on which to request changes to their supplier contracts and more information about the way in which their computer programs work. However, APP entities will be dependent on suppliers offering “clear and high-level” information about how their software can be used to make decisions, in order to meet their own transparency obligations under APP 1.8. Particularly where a supplier is based outside of Australia, the supplier may not be comfortable offering this level of information to what is ultimately a small customer base. We recommend requesting such information in a considered and strategic way, to increase the likelihood of obtaining something useful.

How much information should APP entities include in their privacy policy?

The Fact Sheet gives, in Examples 6 and 7, template disclosures which are designed to comply with APP 1.8. The OAIC is clear that commercial-in-confidence information does not need to be disclosed – this is also stated in the EM and, given the high-level nature of most privacy policies, is unlikely to be a concern for many APP entities. However, for those who are relying on computer programs developed in-house, the question is more complex. Social media companies may use tools to automatically moderate user content. While Example 6 suggests that these companies would not need to disclose how they weight different data points to flag content which breaches their terms of service, they may still wish to avoid disclosing the personal information on which such tools were trained, in order to avoid individuals finding ways to contravene such tools. APP 1.8 expressly requires the disclosure of “the kinds of personal information used in the operation of computer programs”, so it is not clear that they can avoid doing so.

As always, the Guidelines emphasise the importance of “striking an appropriate balance between information which is articulated clearly and in plain language, but also appropriately tailored to be meaningful…while avoiding overwhelming levels of detail.” Example 7 shows how difficult this is in practice. References to a government insurance agency using personal information (like education history and local information) to “support decision-making, administrative action [and] workflow automation” is unlikely to provide the average Australian with a meaningful, privacyforward understanding about how their personal information is used in the context of granting support payments and training programs.

What should APP entities be doing now?

  1. Map computer programs, but also the decisions made using them. While the term “computer program” is broad, the application of APP 1.7-1.9 is still caveated, so understanding both the computer programs which are used to support decisions, but also how they are used (and the kinds of decisions made) is critical.

  2. Record decisions made and assess whether they meet the threshold in APP 1.8. This will depend on various factors, including the personal information used, the way in which the information interacts with the computer program, how output from the computer program is used and whether and how a human is involved in the process. If your intention is to avoid disclosure under APP 1.7-1.9, consider (and document!) how humans can meaningfully challenge the decision, including through avoiding reliance on output from computer programs.

  3. Consider third-party contracts. If your organisation is caught by APP 1.7-1.9, now is the time to request information from your third-party providers that will allow you to meet your obligations under APP 1.8. Use the right tactics to compel provision of this information and consider providing suppliers with a sample disclosure for them to review and revise.

  4. Update privacy policy. The challenge for many organisations will be to strike a balance between disclosures that are overly technical, but which are specific enough to be meaningful to the average Australian. For multinational organisations, disclosures made under Articles 12-14 of GDPR, or “clear and conspicuous” notice issued under US state law, could be a good starting point.

  5. Prepare for consequences. The OAIC has identified a potential uptick in privacy complaints arising from this change to the privacy law. Even though APP 1.7-1.9 does not grant individuals a right to object to automated decision-making, it is likely that many will use it as a basis to seek more information about a decision or, at worst, challenge a decision which is not made in their favour. Be wary of relying on automated decision-making in assessing whether and how to respond to an individual’s privacy complaint, as that is likely to fall within APPs 1.7-1.9 and perpetuate further disclosures in your privacy policy.