Very Difficult to Track or Control: Australia's Privacy Regulator on Personal Information and Commercial AI Products, October 2024

On October 21, 2024, Australia's privacy regulator, the Office of the Australian Information Commissioner, published guidance on privacy and the use of commercially available AI products. The page says it was updated on January 17, 2025. It is aimed at organizations deploying AI systems that were built with, collect, store, use or disclose personal information, and it says the Privacy Act 1988 and its Australian Privacy Principles, the APPs, apply to all uses of AI involving personal information. An organization covered by the Privacy Act, it says, will need to understand its obligations under the APPs when using AI. It also addresses AI products that are freely available, such as publicly accessible chatbots. It is "not intended to be a comprehensive overview of all relevant privacy risks and obligations," it says, and it does not address considerations from other regulatory regimes. Parts of it follow, not the whole.

"AI systems are known to produce inaccurate or false results," the guidance says, and that risk "is especially high with generative AI." It also says a number of uses of AI are low-risk. The use of personal information in AI systems, in its words, is "a source of significant community concern and depending on the use case, may be a high privacy risk activity."

What Goes Into a Prompt

Entering personal information into an AI system, the guidance says, may be a use of that information, if the data stays within the organization's control. Or it may be a disclosure, if the data is made accessible to others outside the organization and released from its effective control. Either way it must comply with APP 6. Under that principle, personal information "can only be used or disclosed for the purpose or purposes for which it was collected" or for a secondary purpose if an exception applies. The common exceptions the guidance names are the individual's consent, and a secondary purpose the individual would reasonably expect that is related to the primary purpose, or directly related for sensitive information.

One example in the guidance is an insurance company. Its employees experiment with a publicly available AI chatbot, enter the details of a customer's claim, including personal and sensitive health information, and ask it to prepare a report assessing whether to accept the claim. By entering the information, the guidance says, "the insurance company is disclosing the information to the owners of the chatbot." It may be difficult to show that customers reasonably expected that disclosure if they were not specifically notified of it, "given the significant public concern about the privacy risks of chatbots."

"As a matter of best practice," the OAIC recommends that organizations "do not enter personal information, and particularly sensitive information, into publicly available generative AI tools, due to the significant and complex privacy risks involved." Once personal information has been input into AI systems, particularly generative AI products, the guidance says, it "will be very difficult to track or control how it is used, and potentially impossible to remove the information from the system."

Who Else Can Read It

The guidance asks organizations to review carefully the terms and settings that apply to their use of a product, and to understand whether the service terms give the developer access to data the organization inputs or generates. Some commercial AI products, it says, include terms or settings that let the product owner collect the data customers input for further training and development of AI technologies. This "can create privacy risks," it says, "with the potential for personal information input into an AI product then surfacing in response to a prompt from another user."

Where a product's operation or features require disclosure of personal information to the developer or third parties, an organization will need to consider whether it is appropriate to proceed with using the product, and if so what measures to place around its use. The measures it says could be relevant include turning off features that would disclose personal information, giving any required notice to the individual, controls to prevent personal information from being entered, and prohibiting staff from entering it. Any such controls or prohibitions, it says, "will need to be accompanied by robust training and auditing measures."

Wrong, and Still Personal Information

Information "can be personal information whether or not it is true," the guidance says, including false information generated by an AI system, such as hallucinations or deepfakes, where it is about an identified or reasonably identifiable individual. Under APP 10, organizations have an obligation to take reasonable steps to ensure the personal information they collect, use and disclose is accurate. Organizations must take those steps, it says, "commensurate with the likely increased level of risk in an AI context."

Its practical tips ask for records that "clearly indicate where information is the product of an AI output and therefore a probabilistic assessment rather than fact," along with the data and system used to generate the output. It is important, the guidance says, that a human user "should be responsible for verifying the accuracy of any personal information obtained through AI, and can overturn decisions made." Staff should be trained, and have the resources, to verify input data, "critically assess outputs," and give meaningful explanations for decisions to reject or accept what the system produces.

Using AI in decisions that may have "a legal or similarly significant effect on an individual's rights" is, in its words, "likely a high privacy risk activity." Its examples include AI software that makes predictive inferences about a person to decide a client's insurance claim or home loan application, or to provide financial advice.

Not Set and Forget

Due diligence for AI products, the guidance says, "should not amount to a 'set and forget' approach." Regular reviews of the product's performance, staff training and monitoring should be conducted "throughout the entire AI product lifecycle." The ongoing processes it lists include internal policies that define permitted and prohibited uses, clear processes for human oversight and verification of AI outputs, "regular audits or monitoring of the output of the product and its use by the organisation," and training on the specific product, its limitations and its risks.

Another of its examples is an AI assistant that staff find very useful, and that produces more errors over time as its underlying processes and training data become outdated. Unless the organization has "robust processes in place to provide human oversight and review of these outputs," ensures the tool remains fit for purpose and trains staff to understand its limitations, the guidance says, "there is a risk that staff will come to over-rely on the assistant and overestimate its accuracy." AI products, it says elsewhere, "should not be used simply because they are available."

The advice against public chatbots is framed as best practice. The record that marks an AI output as a probabilistic assessment, and names the system behind it, sits among the practical tips for APP 10. For a firm whose staff prepare client work with AI tools, the second may be the harder test: could its files show which statements about a client came from an AI output, and from which system?