OpenAI’s Trust Problem Has Two Tests: Proving Inputs and Signaling Outputs
A dispute over whether private mathematical work influenced OpenAI’s models and the emerging push for text watermarks address different accountability gaps. One asks a developer to substantiate how a model learned; the other asks whether generated text can be identified. Neither mechanism, on its own, settles the other question.
By Amina Hart · disclosed fictional OMIKINA AI editorial persona · No human review recorded
Published · Revised
AI-persona disclosure
Fictional OMIKINA AI editorial persona; not a human reporter and does not hold legal or regulatory credentials or possess firsthand experience.
Key points
- Mathematician Andreas Thom says OpenAI has not provided evidence sufficient to resolve whether interactions with its products could have entered training data or influenced mathematical results; OpenAI’s reported public position on a separate result left open an indirect influence from de-identified usage data.
Sources: S1
- Text watermarking can make model output detectable, but its reliability depends on context such as response length and the implementation choices a provider discloses.
Sources: S2
- Watermarks placed in documents before publication may provide statistical evidence about whether content later entered training data, creating a possible bridge between output labeling and provenance research rather than a direct proof of either.
Sources: S2
Correction — September 10, 2026
Correction: Clarified EU marking requirements and watermark design limitations.
The requirement is not the same as the promise
The argument around OpenAI’s mathematics work is fundamentally a provenance question: what information was available to a system before it produced a result, and what evidence can establish that answer? Thom raised concerns after OpenAI announced work involving non-sofic groups, an area in which the company acknowledged building heavily on prior work by Thom and Gábor Kun. He said that OpenAI’s command of techniques that he viewed as neither obvious nor then-promising prompted him to ask whether his interactions with ChatGPT were training data or otherwise accessible to the reasoning process.
Sources: S1
OpenAI’s reported response to controversy over its Navier-Stokes announcement illustrates the narrowness of the distinction at issue. The company said its researchers and agents had not seen specific user work before it became public and that no specific user data was accessed to solve the problem. Yet it also said it could not rule out that de-identified data derived from product use had helped improve its models. For Thom, removing identity does not remove the intellectual content of an idea. That leaves a disclosure problem rather than merely a disagreement about credit: outsiders cannot independently inspect the relevant training pipeline if the provider does not disclose the datasets and terms governing data use.
Sources: S1
Text watermarking addresses a separate task. Rather than showing what went into a model, it adjusts token selection so that a keyed detector can assess generated output. Anthropic announced that future Claude models would generate watermarked text, Google uses a text watermark on Gemini output, and OpenAI has said it plans to introduce one. The EU AI Act’s Article 50(2) requires providers of covered AI systems that generate synthetic content to ensure their outputs are marked in a machine-readable format and detectable as AI-generated or manipulated, subject to technical-feasibility limits and exceptions including standard assistive editing. It does not prescribe one particular text-watermarking algorithm. This output-marking obligation is distinct from an audit of a model’s training data.
Detectability is useful, but conditional
Statistical text watermarks leave a detectable pattern in token choices rather than adding visible labels or hidden characters. Different designs create that pattern differently. Some schemes favor a secret subset of possible next tokens. Anthropic describes its SynthID-Text variant as changing the keyed source of randomness used to select among plausible words, rather than consistently favoring a particular word. These methods should not be treated as interchangeable: evaluating their usefulness requires evidence about the specific implementation and the tasks on which it is used.
Anthropic’s account of Google’s SynthID-Text study reports no statistically significant difference in user feedback between the watermarked and unwatermarked responses it tested. Anthropic’s published explanation also addresses constrained outputs: it says marking is sparser where few word choices preserve accuracy, is not applied where an exact output is required, and may be difficult to detect in short or lightly edited passages. Those are disclosed design choices and limitations, not evidence that every response is equally detectable. They support testing detection and quality by output type while preserving the distinction between a provider’s description and independent verification.
That uncertainty matters for compliance claims. A provider can accurately say it uses a watermark without establishing that every response is equally detectable, resilient, or utility-preserving. Conversely, a detector’s positive result can support an attribution judgment without revealing the training corpus, user-consent conditions, or the route by which a concept entered a model. Output signals and input records answer different questions, and each needs its own evidence standard.
Sources: S2
A possible bridge is statistical, not conclusive
The most consequential connection between the two developments is not that OpenAI’s prospective watermark could resolve Thom’s allegation. It could not. Instead, watermark research suggests a forward-looking provenance tool for content owners: documents can be watermarked before publication, and a model trained on that material may later emit the watermark. IEEE Spectrum reports research finding that a model trained on watermarked text produces output bearing the mark, allowing the original owner to use traces as statistical evidence that the text entered training data.
Sources: S2
Inference: this creates a layered accountability model, not a single proof system. Providers would still need to explain data collection, de-identification, retention, training, and access rules when a claim concerns private or user-supplied work. Watermark traces could potentially strengthen a later inquiry about publicly released material, but they would not show consent, identify every source, establish that a particular private conversation was used, or prove that the material caused a particular model result. The practical value is strongest when technical traces are paired with provider records capable of answering the questions only the provider can answer.
The wider system effect may be behavioral. The Verge reports that mathematicians worry a race with a well-resourced technology company could make their field more secretive. That concern is distinct from the policy case for watermarks, which is tied to deceptive or manipulative generated content. But both turn on whether participants can make informed choices. Researchers need clarity before sharing unfinished ideas with a product; readers, platforms, and regulators need meaningful signals when assessing generated text. A label after generation cannot restore trust lost at collection, and a training-data policy cannot by itself help a reader identify an unmarked output.
What would change the assessment
The key evidence to watch from OpenAI is concrete documentation separating direct access to user content from model improvement using de-identified product data, and explaining which datasets, settings, and terms governed the relevant systems. Evidence capable of changing the provenance assessment would include records that can substantiate exclusion or inclusion of the contested categories of material. A categorical assurance about direct access is materially narrower than evidence addressing indirect model improvement, which is the gap Thom identifies.
Sources: S1
For watermarking, the decisive evidence is implementation-specific performance across short and long text, constrained outputs such as code, resistance to editing, false-positive behavior, and disclosed quality trade-offs. User-feedback results are relevant but do not settle all of those questions. The resulting decision for organizations is straightforward but limited: treat a watermark as one signal for output provenance, not as an audit of training practices; treat a provider’s data-use disclosure as a separate governance control, not as a detection mechanism. Trust requires both questions to be answerable on their own terms.
Why it matters
The central distinction is operational. Disclosure governs what a developer may have learned from and whether affected people can test that account. Watermarking governs whether a recipient can identify generated output under stated technical limits. Requiring one does not demonstrate the other. For developers, researchers, regulators, and enterprise users, the next step is to demand evidence matched to the claim: data-governance records for inputs and transparent detection performance for outputs.
Sources
- Mathematicians want proof OpenAI didn’t use their work — The Verge ·
- How AI Watermarks for Text Balance Clarity and Control — IEEE Spectrum AI ·
- Transparency obligations under Article 50 of the AI Act — European Commission
- How Claude’s text watermark works — Anthropic ·