Designing for Children’s Autonomy and Agency in the Age of Artificial Intelligence

While AI systems offer huge potential for new ways of learning, socializing, or staying connected, they also pose various risks to children [14], many with lasting effects.

• Unfair outcomes expose children to discriminatory or inaccurate information that may arise from AI systems when poor-quality datasets are used to train AI models or when incorrect and biased assumptions are embedded in their design. While such risks are widely recognized [15], they can have particularly profound effects on children, including the denial of essential social care, educational, and economic opportunities [16], and equitable treatment within the justice system [17]. Children are often positioned as passive subjects about whom decisions are made, with limited agency to understand, contest, or influence these determinations [18].
• Harmful content and activities can be generated by increasingly accessible AI techniques, contributing to cyberbullying, misinformation, or criminal behavior [19]. The personalization mechanisms of many AI algorithms may further amplify these risks by reinforcing “echo chambers,” increasing children’s exposure to misleading information, and contributing to adverse mental health issues [20]. Children are typically afforded minimal transparency or agency to control or configure how information is curated and presented to them [21]. There is also a lack of widely accessible mechanisms for children to seek assistance or guidance [18], disproportionately harming those in vulnerable circumstances [20], [22].
• Data privacy is frequently compromised by the pervasive data collection, processing, and inference practices underpinning contemporary AI systems [5]. These systems construct highly personalized profiles of children to predict their actions and interests, thereby influencing patterns of engagement and behavior. Such practices can inflict both immediate and enduring harm on children’s capacities for critical reflection, autonomy, and self-regulation [23]. ”Privacy paradox” [24] presents a recognized challenge for adults, and children’s comparatively limited cognitive and socio-emotional development renders them even more susceptible to the adverse effects of datafication [25]. The opacity of datafication, coupled with insufficient mechanisms that enable children to exercise agency or informed consent, exacerbates their vulnerability to AI-driven technologies and platforms, diminishing their self-determination and impeding their overall well-being. Adolescents are disproportionately affected because of their heightened need for identity formation and peer approval [23].
• Attachments to AI systems can emerge when these technologies are designed with features to capture and sustain users’ attention through persuasive, gamified, or habit-forming interaction strategies, or when they exhibit anthropomorphic features intended to elicit emotional engagement [26], [27]. Such design choices can be particularly concerning for vulnerable children [22], as they exploit their developing cognitive and socio-emotional capacities and pose direct challenges to children’s rights to agency. Current AI designs provide limited scaffolding for children to seek guidance or to develop self-regulation skills and competencies necessary to navigate these human-machine relationships [28].The resulting dependence can produce immediate detrimental effects [22], [29] and long-term consequences for emotional resilience and autonomy [26].

Prioritizing agency provides a critical lens for developing more ethical AI solutions for children that respect their voices, consider their unique developmental needs, and ensure that design choices are made thoughtfully.

While these potential risks have received varying levels of attention in policy and research, the role of children’s agency has often been overlooked. Prioritizing agency provides a critical lens for developing more ethical AI solutions for children that respect their voices, consider their unique developmental needs, and ensure that design choices are made thoughtfully. In this way, we are more likely to create protective measures that are more balanced, providing opportunities for children to develop resilience and self-regulation while avoiding overly restrictive designs that undermine their choice and autonomy.

Agency is closely related to the concept of autonomy. Autonomy is generally understood as a person’s ability to act in alignment with one’s own beliefs, values, and motivations [30]. This definition suggests that autonomy can relate to the capacity to act on them and the authenticity to act on one’s beliefs, values, and motivations. A widely accepted definition of agency is the ability to make meaningful choices [31]. This definition reflects that agency means more than mere control but also the capacity to act toward internal motivation, critical thinking, and problem-solving. Scholars argue that making effective choices requires the ability to set goals (or intentions), exercise self-control and self-monitoring, and reflect on progress to adapt strategies and sustain motivation [32]. This shows that agency is a multifaceted concept and requires careful consideration in digital designs.

In addition to considering agency as an individual capacity, insights from psychology extend this perspective by showing that people’s regulation of motivations and actions is not only governed by inner reasoning processes but also influenced by their environment [31]. This view situates agency as a social construct, recognizing that the human mind is not only reactive but also “generative, creative, proactive, and reflective” in response to external stimuli [31]. Accordingly, it is important to consider agency in forms beyond direct personal agency [31] but also in relation to others, through lens such as proxy agency, which occur when individuals rely on others who hold resources, expertise, or power to act on their behalf; co-agency, which refers to cases where an individual’s ability to act is co-developed with or supported by others; and collective agency, which arises when people work together to achieve shared goals that none could accomplish alone. These relational lenses of agency provide a complementary and critical addition to personal agency.

Developing a clearer framework for understanding children’s agency is critical to ensure that discussions of children’s autonomy and agency are developmentally appropriate.

The concept of agency for children requires some more specific considerations and has been given further nuances by different academic disciplines. In philosophy, it is commonly recognized that children may not possess the same level of agency as adults [33], even though they exhibit increasing autonomy and decision-making abilities as they grow. However, agency is considered crucial for children’s knowledge acquisition in learning sciences, central to supporting children’s learning [34]. In development psychology, literature further highlights how children’s agency can be influenced by their goal setting [32], epistemic abilities [35], such as prior knowledge about a specific topic, and cognitive development stages [36], which may affect their metacognitive skills in evaluating current situations and making decisions.

This plurality of agency conceptualization for children means that clarity is critical to effectively discuss their roles in addressing potential harms and risks from AI systems for children. For example, policies calling for protection from AI-facilitated manipulation and deception assume the existence of inner governance abilities (whether in the formative stage, such as childhood, or in a more mature stage, like adulthood), while those emphasizing the importance of retaining control over one’s decisions (e.g., in the context of harmful content or persuasive design) assume the presence of capacity and agency. However, the gap between the assumptions made in policy frameworks and the forms of agency that AI systems actually afford to children may lead to mismatches between expected benefits and persistent harms. Developing a clearer framework for understanding children’s agency is critical to ensure that discussions of children’s autonomy and agency are developmentally appropriate. This can help avoid over-protecting children by assuming weak agency and thereby constraining their development, or overlooking their vulnerability by assuming full capacity.

Designing for children’s agency is fundamental and complementary to other ethical considerations for designing AI systems for children. Research has identified significant gaps in how ethical concerns are addressed in existing AI systems that support children’s learning, health, and safety [18]. In particular, considerations like data privacy and safeguarding are frequently overlooked, even in systems designed to create a better digital service for children. A separate review of ethical AI policies and regulations similarly found a lack of child-specific focus [37], leaving children’s unique vulnerabilities unaddressed. These gaps highlight the need for child-specific AI ethics principles. Both reviews also highlighted the lack of consideration of designing for children’s agency, both in practice and in existing ethical AI frameworks. This is a critical omission, given that current ethical discourse on AI tends to frame protection as the main response to algorithmic harms. While protection is essential, an over-reliance on protective measures risks diminishing children’s autonomy and participation.

Ethical principles, such as fairness, transparency, accountability, privacy, and safety, are all intended to uphold children’s best interests and digital rights, as articulated in the United Nations Convention on the Rights of the Child (UNCRC) [38]. Yet, the principle of designing for children’s agency presents a critical opportunity to ensure both a focus on safeguarding children and fostering their ability to act, decide, and understand. Without deliberate attention to agency, AI systems designed for children may risk curtailing autonomy, diminishing critical thinking, and undermining the very rights they are meant to protect.

We argue that designing for children’s agency is not only fundamental to all other ethical principles but also complementary to them. Agency should not be treated as an additional or optional ethical principle, but as a foundational lens through which all other principles acquire meaning [37]. Children neither demand fairness from an AI system they cannot interrogate nor can they benefit from transparency without the confidence or skills to interpret what is disclosed. The agency operationalizes these principles, transforming them from abstract commitments into lived experiences. For example, it supports transparency by enabling comprehension, underpins accountability by encouraging questioning, and enhances fairness by empowering users to contest outcomes. By considering agency along with other ethical principles, we can expand current ethical approaches by including concrete mechanisms that foster informed choice (along with transparency), intrinsic motivation (for accountability), and reflective engagement (along with contesting fairness). In doing so, AI systems can help children not only remain protected but also develop the ability to understand and act on principles like transparency, accountability, fairness, privacy, and safety. This shifts AI ethics for children from a protection-based model toward one grounded in empowerment and participation.

Positioning agency as fundamental in the design of AI systems for children means prioritizing age-appropriate mechanisms that encourage active engagement and collaboration among children, families, and caregivers, while enhancing their user experiences and access to meaningful knowledge and support.

Positioning the agency as fundamental in the design of AI systems for children means prioritizing age-appropriate mechanisms that encourage active engagement and collaboration among children, families, and caregivers, while enhancing their user experiences and access to meaningful knowledge and support. Children’s sense of agency evolves rapidly throughout childhood; so, fostering agency must begin in early childhood. A plethora of AI systems [39] are explored to support children’s early childhood language and literacy development [40], enquiry skills [41], or communication [41]. The design of such systems should be guided by principles that promote self-regulation, balanced engagement across technologies and relationships, and the freedom to explore and learn safely so that we avoid creating digital divisions for parent-child co-engagement or undermining children’s agency development with accidental manipulations [42].

Supporting children to become more autonomous in the digital context has attracted growing attention in recent years [43]. While such support is needed across multiple aspects of AI systems, ranging from ethical algorithmic design to inclusive training of AI models, a child-centered approach remains critical to ensure that technical solutions are guided by children’s needs and voices. However, in the absence of consensus on what autonomy and agency entail for children in digital contexts, design strategies have been diverse. A recent landscape review identified four primary mechanisms for supporting children’s digital autonomy, including contextualization, nudging, peer support, and scaffolding [44]. Contextualization addresses the challenges of helping children grasp abstract concepts by embedding them in familiar, fictional or meaningful contexts; nudging draws on the concept of choice architecture [45] to shape behaviors; peer support is particularly crucial in early childhood and adolescence, to support children learning new behaviors by observing, modeling, and reinforcing the behaviors of others; and scaffolding is rooted in theories related to children’s knowledge acquisition process, such as Vygotsky’s zone of proximal development [46], to help children make their own decisions by building on the knowledge and skills they already hold.

While these design approaches all have the potential to affect children’s inner governance, capacity, and motivation development, their focuses on agency differ. For example, contextualization primarily strengthens cognitive understanding, nudging tends to support children’s intrinsic motivation and behavioral change, peer support builds on social influence, and scaffolding often focuses on children’s self-regulation. These differences highlight the multifaceted nature of agency, which involves cognitive, motivational, reflective, and choice-making processes through which individuals are able to act intentionally and achieve effects.

While these mechanisms represent important strides toward fostering individual agency, they largely neglect the relational dimensions of agency, that is, the ways children’s decisions, understanding, and interactions are shaped by relationships with peers, caregivers, and technology. Current child-centered research has predominantly addressed how support children acting independently, but has given comparatively little attention to how agency is co-constructed, negotiated, or constrained in relation to social and technological contexts. This gap limits opportunities for designing AI systems that can support children not only as independent actors but also as socially embedded learners whose choices and growth depend on relational support.

Agency provides the connective tissue between existing ethical principles, the bridge between wellbeing and resilience, and the foundation for meaningful participation in digital life.

Different social contexts give rise to different forms of agency, and recognizing these distinctions is critical for designing digital systems that support the right kind of agency for the right needs. For example, to protect children and adolescents from AI-facilitated manipulation and deception, we should not only support their inner governance by fostering self-regulation but also consider the peer groups and related social contexts that shape their decisions. In this way, we could draw on teenagers’ intrinsic desire for identity and peer recognition by creating designs that promote co-agency that de-emphasize quantitative metrics for peer approval (e.g., removing “likes”) and promote expressions that prioritize appreciation, empathy, and shared interest (e.g., “I relate” and “That’s thoughtful”). We should also support adolescents’ ability to seek guidance and navigate social conflicts by making help-seeking visible, voluntary, and configurable. While a wide range of strategies have been developed to support children’s individual self-regulation and agency, far less attention has been given to integrating the broader social contexts that influence these capacities, apart from mechanisms such as peer support. This gap highlights an important direction for future work: children’s digital agency should be understood not only as an individual capacity but also as something shaped, constrained, and enabled by the social environments in which they act.

The design of ethical AI cannot be adequately addressed through protection alone [47], [48]. Agency provides the connective tissue between existing ethical principles, the bridge between wellbeing and resilience, and the foundation for meaningful participation in digital life. To neglect agency is to risk undermining the very aims that ethical AI seeks to achieve. Designing with agency at the center is to create systems that not only respect children’s rights but also equip them to grow, decide, and flourish in an AI-mediated world.

While the child-centered research has extensively explored ways to foster children’s agency, much of this work has emphasized children’s individual agency over its social dimensions. This imbalance is critical to address, given that children’s development involves diverse social contexts. Moreover, as children’s interactions with AI systems become increasingly relational, design approaches must ensure that such systems do not isolate children from their physical and social worlds, or hinder their rights to creativity, play, and flourishing. There is an evident gap not only in the current child-centered AI research but also within the current AI policy and governance frameworks. AI becomes increasingly ambient and embedded in everyday life, raising new challenges to autonomy and decision-making, making agency a priority across all demographic groups. The ethical design of an AI system for children is an imperative and a responsibility. Prioritizing children’s agency in the design of these systems is essential to supporting their empowerment, resilience, and capacity to flourish in an AI-saturated future.