The Widespread Adoption of AI-Powered Learning Devices and Their Potential Security Risks

In recent years, AI-powered learning devices have rapidly gained popularity worldwide, becoming an essential tool in many households. Statistics show that over 90% of children’s learning devices are now equipped with large AI models. These advanced models significantly enhance the interactivity between the device and children. For instance, they can serve as intelligent reading companions, automatically grade assignments, and answer questions, making them a valuable supplement to traditional school education. With such benefits, AI-powered learning devices are becoming an integral part of home education.

However, while we recognize these technological advancements, we must also be aware of the underlying security risks. Many of these devices, equipped with large language models (LLMs), have not undergone rigorous security testing and reinforcement, leading to several potential safety concerns.

Firstly, there are significant privacy concerns. AI-powered learning devices require vast amounts of data from children to optimize and personalize their learning experience. This data may include sensitive information such as children’s voices, text inputs, learning habits, and even emotional responses. If this data is accessed or misused by malicious actors, it could lead to severe privacy breaches. Parents may not fully understand how these devices collect, store, and use this data, nor can they guarantee that the data won’t be used for unauthorized purposes.

Secondly, the content filtering and monitoring capabilities of AI learning devices have limitations. While these devices can customize content based on a child’s age and learning level, they are not always reliable in screening and filtering inappropriate material. The content generated by large models can sometimes include biases, inaccuracies, or even harmful content. If children inadvertently encounter such information, it could mislead them or lead to misconceptions.

Additionally, the algorithms of AI learning devices themselves may have security vulnerabilities. Most AI learning devices are designed primarily to enhance learning efficiency rather than to withstand external attacks, making them susceptible to cyberattacks. If these devices are hacked, it could not only disrupt children’s learning but also compromise the security of the household network.

The darkside of technology: risks of LLM content security

Currently, many AI learning machines are connected to the Cloud LLM API through wifi or 5G networks, and the received prompts are directly input into the fine-tuned domain LLM hosted in the cloud.

The attack surface here can be summarized as follows:

• Intrinsic security jailbreak risk: After fine-tuning and aligning the domain data (such as educational data), its intrinsic security defense capability may be weakened.

• System Prompt Jailbreak: The App inside the learning machine is essentially an AI Agent, and some System Prompts have been set up during the development stage, such as “Book Companion Reading” and “Exam Paper Correction”. However, attackers can break through the limitations of System Prompt through social engineering, linguistic deception and other means, and guide LLM to produce content outside the design scope.

Now from the technical level discussion, back to the real world, a possible real risk is that：

Many parents buy these smart devices for their children to learn, but once minors learn these jailbreak technologies out of curiosity, they break through the original software design goals through Jailbreak, and use the learning machine to achieve purposes other than learning, such as surfing the Internet, generating violent and pornographic content, causing children’s physical and mental health to be polluted.

A simple demo

Note：The following cases are only for technical and learning purposes and do not represent my own views.

Disclaimer: The following conversations and examples were intended for research purposes.