Generative AI at work: the risks that rarely make the slide
Lack of transparency, no accountability, mistakes at scale and costs nobody sees coming
In July 2025, the Cologne Local Court in Germany received a brief in a family law case that looked flawless at first glance. It cited commentaries, monographs and law journals. The court checked and noted that “the cited sources, too, are entirely made up” (our translation). In the court's assessment, the arguments had apparently been produced with the help of artificial intelligence.
The case is no outlier. It shows what generative AI does best – and why exactly that becomes a problem in the office: it produces text that sounds convincing. Whether it's true is another matter.
This isn't an argument against AI. We build it into our own products. But anyone who wants to use generative AI in the office should know its risks. Four of them are underestimated particularly often.
1. Lack of transparency: why this answer?
A language model calculates which word is most likely to come next. That produces fluent sentences, but no traceable reasoning. Where a statement comes from often can't be said, and if the model does name a source, that source may be invented, too.
This became widely known in 2023 with the case of Mata v. Avianca in New York. Two lawyers had filed a brief citing six court decisions that ChatGPT had made up. The court imposed a $5,000 fine. The Cologne case shows that the pattern is repeating itself in Germany.
2. No accountability: AI isn't liable
An AI can't take responsibility for anything. It doesn't sign, isn't liable and never has to answer to a bar association. Responsibility stays with whoever uses its results.
Air Canada learned this in 2024. The airline's chatbot had explained a refund rule for bereavement fares to a customer that didn't apply that way. Before the Civil Resolution Tribunal in British Columbia, the company argued that the chatbot was, in effect, responsible for its own statements. The tribunal disagreed: Air Canada is liable for all information on its website, whether it comes from a static page or from a chatbot.
For professionals, there's more: the duty of careful review can't be handed over to software. In its guidance on the use of AI, the German Federal Bar explicitly points to review obligations and to professional secrecy.
3. Mistakes at scale
A person who makes a mistake makes it once. A prompt that contains a mistake makes it in every document it produces. Imagine a template for client letters that calculates a deadline incorrectly – and 300 letters that go out that way. Each one looks plausible, and that's exactly what makes the error so hard to find.
A case at Deloitte in Australia showed in 2025 that large firms aren't immune either. A report for the Australian Department of Employment and Workplace Relations, for which around AU$440,000 was paid, contained invented references and a fabricated quote from a court judgment. Deloitte confirmed it had used a generative AI tool based on GPT-4o in preparing the report, revised it and refunded part of the fee. Whether the AI caused the errors, the company left open.
4. Costs nobody sees coming
Generative AI is usually billed by usage, per request and per amount of text processed. What costs a few euros in a test can quickly multiply in everyday use, especially when AI agents work on their own in loops. On top of that, the provider can change prices and terms.
Back in 2024, the research firm Gartner predicted that by the end of 2025, around 30 percent of generative AI projects would be abandoned after the proof of concept. The reasons Gartner named: poor data quality, inadequate risk controls, escalating costs and unclear business value.
And where does the data go?
One risk cuts across all four: the flow of data. Anyone using a cloud model sends their input to the provider's servers. For law firms and other professionals bound by secrecy, that's delicate, because breaching confidentiality is a criminal offense in Germany (Section 203 of the German Criminal Code). Models running on your own hardware solve this problem, because the data never leaves the house. Cloud models, on the other hand, are often more capable. It's a trade-off, and it should be made deliberately.
What the EU AI Act requires
Since February 2, 2025, Article 4 of the EU AI Act has required providers and deployers of AI systems to ensure a sufficient level of AI literacy among their staff. With the so-called Digital Omnibus, the EU softened the provision in summer 2026: a specific level of competence no longer has to be guaranteed. The obligation to take measures that promote AI literacy remains, though. So if you use AI in the office, train your team and document it.
Where generative AI belongs
After so many risks, this may sound surprising: generative AI is still useful. It helps with a first draft that a person then reviews. It summarizes long documents, as long as it names where it found things. And it drafts routine replies that someone reads before they go out. The common thread: a person checks the result before it counts.
Sometimes, though, the best AI is no AI at all. Many tasks that generative AI is promoted for today can be solved more reliably when data is structured. A value that was entered correctly once, and whose origin is known, doesn't need to be guessed by a language model.
Rules of thumb for everyday office work
- Use generative AI where mistakes are cheap and visible.
- Check every source, every figure and every quote before it leaves the house.
- Don't put confidential data into cloud services before you've clarified the consequences.
- Keep an eye on costs, especially in automated workflows.
- Train your team and document who uses which AI for what.
How we use AI
In our suite, LangAux is the platform for generative, language-based AI, and it's model-agnostic. You can run a model on your own hardware, in which case no data leaves your systems. We take care of selecting and setting it up, so you don't need in-house AI expertise. Or you connect a cloud provider such as Anthropic (Claude), OpenAI or Google (Gemini). These models can be more capable, but your data then leaves your systems, and the provider bills you directly for usage. The choice is yours.
In TaskAux, AI tasks – currently in Early Access – run as a step of their own in the chain. Where it matters, they're followed by a task in which a person checks the result. That way, AI remains a tool, and responsibility stays where it belongs. What analytical AI on structured data looks like is the subject of our next article.
Sources
- beck-aktuell: Cologne Local Court on an AI-generated brief with invented sources (July 8, 2025, German)
- Bloomberg Law: Phony ChatGPT Brief Leads to $5,000 Fine for NY Lawyers
- American Bar Association: BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot
- Accounting Times: Deloitte to refund government after using AI in $440k report
- Technology Decisions: Gartner predicts 30% of GenAI projects to be abandoned
- anwaltspraxis Magazin: German Federal Bar guidance on AI in law firms (German)
- ITMR Legal: The Digital Omnibus on AI (German)