An AI model sharing marketplace, has introduced "bounties" to encourage users to create deepfakes of real people. The use of AI amplifies the realism, and requests targeting private individuals with limited online presence heighten privacy concerns.
(Photo : STEFANI REYNOLDS/AFP via Getty Images)
This photo illustration created on July 18, 2023, in Washington, DC, shows an advertisement to create AI girls reflected in a public service announcement issued by the FBI regarding malicious actors manipulating photos and videos to create explicit content and sextortion schemes.
Deepfake Content of Real People
A marketplace for sharing AI models called 'Civitai' has introduced a controversial feature known as "bounties" to motivate users to create convincing deepfake content of real people. The top-performing AI model creator is awarded with the platform's virtual currency, "Buzz," which can be purchased with real money.
Numerous bounties on the platform focus on replicating the appearances of celebrities and social media influencers, particularly emphasizing female figures. Engadget reported that this sheds light on the majority of these outputs being categorized as "nonconsensual sexual images."
While such content has existed on the internet for years, the utilization of artificial intelligence amplifies the realism of the final deepfake. Requests were found targeting private individuals with limited online presence, heightening concerns surrounding privacy and ethical implications.
Also Read: India Cracks Down on AI Misinformation, WhatsApp May Be Required to Display First Message Source
As reported by Andreessen Horowitz, Civitai currently ranks as the seventh most popular generative AI platform in the market. This implies significant attention and engagement with the bounty requests on the platform.
It's important to highlight that most bounty requests on Civitai avoid explicitly stating the desire for sexual content, often using ambiguous language. Yet, some are explicit, employing terms like "degenerate request" and making comments about female anatomy.
Civitai maintains that bounties shouldn't be utilized for non-consensual AI-generated sexual content of real individuals.
The platform allows for both sexual images of public figures and non-sexual images of private individuals. The concern arises when these two types of images are combined.
Surge in Requests
Since the introduction of the new "bounty" feature on Civitai, the platform has witnessed a surge in requests focusing on replicating the appearances of celebrities and social media influencers, predominantly females. 404 Media uncovered at least one bounty targeting a private individual with minimal online presence, highlighting the potential misuse of personal images.
Michele Alves, an Instagram influencer subject to a bounty on Civitai, shares her concerns about the situation, expressing fear about the implications of image misuse that she hadn't considered before.
Uncertain about the measures she could take, Alves reflects on the uncontrollable nature of the internet and contemplates the potential mental toll of such actions, recognizing the profound impact on her well-being.
This unfolding scenario highlights the increasing accessibility of text-to-image AI tools, the simplicity of generating AI models for specific individuals, and the monetization of nonconsensual content production on platforms like Civitai.
The confluence of these factors raises significant concerns about the generation of nonconsensual images, extending the impact beyond celebrities to affect individuals across the digital landscape.
Related Article: AI-Generated Deepfakes Raise Concerns Ahead of 2024 US Presidential Election
