Legislative regulation of AI-generated abuse content attempts to address the same threats covered in the previous articles (propaganda, phishing, misinformation, hate speech, and deepfakes) through legal rather than technical means. Two jurisdictions have taken notably different approaches. The United States relies primarily on targeted legislation addressing specific harm types alongside voluntary frameworks, while the European Union has implemented broad regulatory frameworks that apply to AI systems and digital services as categories.

This article covers the US and EU regulatory approaches to abuse attacks, including the Take It Down Act, the NIST AI Risk Management Framework, the EU Digital Services Act, and the EU Artificial Intelligence Act.

The liability question

A core issue that legislative regulation must address is liability for LLM-generated content. When an LLM produces harmful output, responsibility could fall on the model developer who built and trained the system, the deployer who integrated it into an application, or the end user who provided the prompt that triggered the output. Different regulatory frameworks assign liability differently depending on the type of harm and the actor’s level of control.

Regulation in this space also needs to avoid restricting the legitimate capabilities that make LLMs useful (education, accessibility, creative work, productivity). Any framework that is too broad risks suppressing beneficial applications. Any framework that is too narrow risks leaving harmful applications unaddressed. The balance between these is an active area of policy debate in both jurisdictions.

Regulation in the United States

In the US, spreading misinformation is generally protected under the First Amendment unless it crosses into defamation, incitement to violence, or fraud. This limits the scope of regulatory intervention. US regulation addresses AI abuse attacks through targeted legislation, voluntary best practices, and existing consumer protection enforcement.

The Take It Down Act

The Take It Down Act (Tools to Address Known Exploitation by Immobilising Technological Deepfakes on Websites and Networks Act) was signed into law on May 19, 2025, and is the first US federal law specifically targeting AI-generated deepfakes. It criminalises the knowing publication of non-consensual intimate images, including AI-generated “digital forgeries”, with penalties of up to two years for adult victims and three years for content involving minors. The law passed both chambers of Congress near-unanimously (409-2 in the House).

The Act also requires covered platforms (websites and applications that host user-generated content) to implement a notice-and-removal process. Platforms must remove reported non-consensual intimate content within 48 hours of receiving a valid removal request. Platform compliance requirements took effect on May 19, 2026, and the first conviction under the Act was issued in April 2026.

The Take It Down Act targets a specific abuse category (non-consensual intimate imagery, including deepfakes) rather than AI-generated abuse content broadly. It does not address misinformation, hate speech, propaganda, or phishing. It also does not regulate the AI tools used to create the content, only the publication and hosting of the output.

NIST AI Risk Management Framework

The NIST AI Risk Management Framework (AI RMF) is a voluntary framework that outlines the characteristics of trustworthy AI systems and the processes for developing and deploying them. It provides guidance on risk identification, measurement, and mitigation across the AI lifecycle. Model developers and deployers who adopt the framework’s recommendations reduce the likelihood of their systems being used for abuse attacks, but compliance is not legally required.

FTC enforcement

The Federal Trade Commission has authority to act against deceptive practices by companies deploying AI. If LLMs are used in fraud, misleading commercial practices, or consumer deception, the FTC can investigate and pursue enforcement. The Take It Down Act also designates platform non-compliance with takedown requirements as a violation of the Federal Trade Commission Act, making the FTC the enforcement body for the platform obligations.

Regulation in the European Union

EU regulation takes a broader approach through two complementary frameworks that apply to digital services and AI systems as categories rather than targeting specific harm types.

Digital Services Act

The Digital Services Act (DSA) requires digital service providers to implement mechanisms for reporting and removing illegal content. Providers must establish reporting systems that allow users to flag illegal material and must respond to reports with appropriate action. An appeal system is required for users whose content was incorrectly removed. The DSA applies to all digital service providers offering services to recipients in the EU, regardless of the provider’s location.

For larger platforms (designated as “very large online platforms” or “very large online search engines”), the DSA imposes additional obligations. These platforms must conduct recurring risk assessments covering issues such as the dissemination of misinformation, cyber violence, and the impact of their services on fundamental rights. Based on assessment results, platforms must implement mitigations such as modifications to recommendation algorithms to limit the spread of harmful content, or increased moderation efforts. These platforms must also disclose their content moderation policies, algorithmic systems, and advertising targeting practices.

The DSA is not AI-specific. It applies to all illegal content on digital services, whether human-generated or AI-generated. However, its reporting, removal, and risk assessment requirements directly address the distribution channels that abuse attacks rely on.

EU Artificial Intelligence Act

The EU Artificial Intelligence Act is a risk-based regulatory framework that applies specifically to AI systems, including LLMs. It applies to providers and deployers of AI applications offering services in the EU, and it classifies AI systems into four risk tiers with corresponding obligations.

Unacceptable risk AI systems are banned entirely. This category includes social scoring systems and AI systems that cause significant harm by employing manipulative techniques, impairing informed decision-making, or exploiting vulnerabilities.

High risk AI systems operate in critical sectors such as healthcare, education, or law enforcement. These face extensive regulatory requirements including risk management systems, data governance standards, and mandatory human oversight.

Limited risk AI systems interact directly with people or generate content. LLMs fall into this category. The primary obligations are transparency and documentation. Service providers must disclose when content is AI-generated and implement safeguards to prevent misuse, including abuse attacks.

Minimal risk AI systems (such as spam filters or video game AI) are largely unregulated.

The AI Act’s classification of LLMs as limited-risk systems means their primary obligation is transparency rather than the extensive compliance requirements imposed on high-risk systems. The transparency requirement that AI-generated content must be disclosed as such directly addresses the detectability challenge of LLM abuse attacks, where the difficulty of distinguishing AI-generated text from human-written text is a key enabler of the threat.

Summary

US and EU regulation of LLM abuse attacks reflects different legal traditions and policy approaches. The US targets specific harm categories through legislation such as the Take It Down Act (deepfake intimate imagery, signed into law May 2025, first conviction April 2026) and relies on voluntary frameworks (NIST AI RMF) and existing enforcement authority (FTC) for broader AI governance. The EU applies comprehensive frameworks across all digital services (DSA) and all AI systems (AI Act), with obligations scaled to risk level. LLMs fall under the AI Act’s limited-risk tier, where the primary obligation is transparency, including disclosure of AI-generated content.