Over 100 AI models on the popular platform Hugging Face were discovered to be malicious, containing code that could steal data or give attackers access to users’ machines. This discovery raises significant concerns about the potential dangers lurking within the seemingly collaborative and open-source world of AI development.

Hugging Face is a platform widely used by researchers, developers, and hobbyists to share, collaborate, and access pre-trained AI models for various tasks, including natural language processing and image recognition. It boasts a vast library of models, fostering innovation and accelerating AI development. However, this open and accessible nature also presents security challenges.

JFrog’s security team, known for its expertise in identifying vulnerabilities, conducted a thorough investigation of the platform. Utilizing advanced scanning systems, they specifically focused on models built with PyTorch and TensorFlow Keras, two popular frameworks used for creating AI models. Their findings were alarming – roughly 100 models harbored malicious functionalities, posing a significant threat to anyone downloading and using them.

Hugging Face, to its credit, already implements various security measures to protect users. These include malware and code scanning, looking for suspicious patterns and vulnerabilities. Additionally, they scrutinize models’ functionalities to identify potential risks like “unsafe deserialization,” a technique malicious actors can exploit to inject harmful code.

However, despite these efforts, JFrog’s research revealed that malicious actors were able to bypass these safeguards. One highlighted case involved a PyTorch model uploaded by a user named “baller423.” This model, since removed by Hugging Face, contained a payload capable of establishing a “reverse shell” connection to a specific server. This essentially allows remote access to the victim’s machine, potentially enabling attackers to steal data, install additional malware, or even launch further attacks on other systems.

The malicious code in this instance exploited a weakness in Python’s pickle module. This module allows objects to be serialized (converted into a format for storage or transmission) and deserialized (converted back into their original form) for later use. The payload used the “reduce” method within the pickle module to execute arbitrary code when the model file was loaded. This effectively disguises the malicious code within a seemingly legitimate process, making it harder to detect.

JFrog further discovered that the same payload used different IP addresses in various instances. This suggests that the operators behind these models might be AI researchers rather than traditional hackers. While their intentions might not be purely malicious, their actions still pose a significant risk. Uploading and sharing models containing vulnerabilities, even for research purposes, can easily fall into the wrong hands and have disastrous consequences.

You may also enjoy reading Unveiling the Dark Side of Malicious Chatbots.

To further understand the intent behind these malicious models, JFrog deployed a “honey pot” – a system designed to attract and analyze malicious activity. While the honey pot successfully established a connection with the malicious model, it was unable to capture any commands sent during the brief period of connectivity. This inconclusive evidence adds to the complexity of the situation, making it difficult to determine the true motives of the individuals behind these uploads.

JFrog emphasizes that these findings are not unique to Hugging Face and highlight a broader issue within the AI community. The potential security risks associated with AI models have not been adequately addressed or discussed, and developers often lack the necessary awareness and training to build secure models.

This incident serves as a wake-up call for the entire AI ecosystem. It underscores the critical need for:

• Thorough security checks: Implementing robust security measures throughout the development lifecycle of AI models is essential. This includes code reviews, vulnerability scanning, and penetration testing.
• Responsible development practices: Developers must be trained on secure coding practices and the potential security implications of their work.
• Increased vigilance: Users need to be cautious when downloading and using pre-trained models, especially from untrusted sources. Verifying the source and reputation of the model creator is crucial.
• Collaboration: The AI community needs to work together to develop and implement comprehensive security standards for model development and sharing. Open communication and collaboration are vital to address these challenges effectively. 

5 Ways End Users Can Stay Safe from Malicious AI Models

1. Be cautious when downloading models: Only download pre-trained models from trusted sources on Hugging Face or other platforms. Check the model creator’s reputation and reviews from other users.
2. Scrutinize model descriptions: Read the model description carefully and look for any red flags, such as vague explanations of functionality or unusual permissions required.
3. Use a sandbox environment: If you must download and experiment with untrusted models, do so in a controlled environment like a sandbox. This isolates the model from your main system, minimizing potential damage.
4. Stay updated: Keep your security software and operating system up-to-date with the latest patches to address any known vulnerabilities that malicious actors might exploit.
5. Report suspicious activity: If you encounter any suspicious behavior while using an AI model, report it immediately to Hugging Face or the platform where you found the model.

It’s important to remember that complete security is never guaranteed, but by following these steps, you can significantly reduce the risk of falling victim to malicious AI models. By taking these steps, the AI community can mitigate the risks associated with malicious actors and ensure that the benefits of AI technology are not overshadowed by potential security threats.