I Built My Daughter an AI I Can't Read
Trust children. Govern the technology.
My daughter was writing a story. She would type a sentence describing what she wanted to happen next, and the AI would produce several paragraphs of prose. Then she’d type another sentence, and another large passage would appear. She seemed perfectly happy with the arrangement, but as I watched, I began to wonder how much writing she was actually doing.
She’s eleven, reads voraciously, and loves writing fiction, including fan fiction based on books she’s enjoyed. She has plenty of ideas of her own, but here she was, gradually becoming a spectator to stories that a machine was writing for her. The AI was doing most of the work: choosing the words, developing the scenes, figuring out how the narrative should unfold. All the little struggles that are part of learning to write were being quietly taken away.
I don’t think there’s anything inherently wrong with using AI to generate fiction. I use AI extensively myself. But I wasn’t sure this was the experience I wanted for a child who was still discovering what she could do with language.
An earlier incident had raised a different concern.
She was preparing a presentation for school and needed to research a topic. I suggested she Google it, as I would have done myself. What followed was an unexpectedly frustrating exercise. The results were full of websites that were either too complicated for her purposes, cluttered with advertisements, or hidden behind paywalls. Even when we found something useful, it was rarely presented in a way that an eleven-year-old could easily understand.
I’ve been using search engines for nearly three decades, and I suppose I’ve developed an instinct for navigating their shortcomings. I know which results to ignore, how to refine a query, and how to extract something useful from a page that seems determined to sell me something first. My daughter doesn’t have that experience, nor should she need it simply to find an explanation for a school project.
Eventually, I suggested we ask an AI together. At least we could phrase the question in ordinary language and get an explanation at a level she understood. We’d still need to check the information, of course, but it seemed a more promising starting point than the search results we’d been struggling with.
These two experiences stayed with me because they presented almost opposite problems. Traditional search was making it unnecessarily difficult for her to find and understand information. Generative AI was making it almost effortless to produce large amounts of content, with very little effort or engagement on her part.
Neither seemed quite right.
I began thinking about what an AI application designed specifically for children might look like. Not a simplified chatbot, or one that merely filtered out inappropriate material, but something that approached learning, curiosity, and creativity differently.
Eventually, I decided to build one myself. I called it Marmalade.
Trust children, govern the technology
Before I got very far, I ran into a question that had less to do with AI and more to do with parenting. If I was going to give my daughter access to an AI system, how much control should I have over what she did with it?
Parental control software generally takes a predictable approach to this problem. It gives parents tools to monitor their children’s activities, sometimes down to the individual messages they send or the searches they make. I understand the reasoning. Parents have a responsibility to keep their children safe, and the internet provides plenty of reasons to worry.
Still, I was uncomfortable with the idea of being able to read every conversation my daughter had with an AI.
Children have questions they may not be ready to ask their parents. They have half-formed ideas, private interests, and occasionally embarrassing curiosities. This is not necessarily something to be alarmed about. It’s part of growing up. I remember being eleven, and I doubt I would have appreciated knowing that somebody could inspect every question I asked or every story I attempted to write.
There’s also a practical problem with surveillance. If children know that everything they type is being read, they may simply become more careful about what they reveal. That doesn’t necessarily make them safer. It may just teach them to take their questions elsewhere.
I wanted to exercise parental responsibility without turning my daughter’s curiosity into something I needed to monitor constantly. The distinction that eventually made sense to me was between governing the technology and monitoring the child.
I should be able to decide which AI models she can use, how much time she spends using them, and how much money the application is allowed to spend on API calls. I should be able to establish sensible boundaries and know when something potentially serious has happened. But I don’t need to read every question she asks to accomplish those things.
That became Marmalade’s guiding principle: Trust children. Govern the technology.
The application has separate parent and child accounts. My dashboard lets me configure time limits, cooldown periods, model access, and spending limits. It shows usage information and limited safety indicators, but has no facility for browsing her conversations. Those conversations are held temporarily and expire after fifteen minutes of inactivity unless she explicitly saves something she wants to keep.
There are safeguards for situations involving serious distress or unsafe contact. The system can provide immediate support information, including relevant helplines, and record certain safety indicators without storing the conversation. For disclosures of abuse or unsafe contact, parental alerts are withheld. The reasoning is straightforward: a safety system shouldn’t risk alerting someone who may be responsible for the child’s predicament.
None of this is a guarantee of safety or absolute privacy. Automated safeguards have limitations, commercial AI providers have their own data-handling policies, and someone with administrative access to a server ultimately controls that machine. I’ve tried to be explicit about these limitations rather than pretend that software can solve every problem of trust and safety.
The important thing, for me, was to build an application in which reading a child’s private conversations wasn’t part of the normal parental experience.
Making room for the child to do the work
I didn’t want to reproduce the standard chatbot interface, with its endless conversation thread and willingness to respond to almost anything. Instead, Marmalade became a collection of distinct spaces arranged around a little treehouse. There’s a Question Room for explanations and general curiosity, a Field Guide for research, a Study Nook for homework assistance, and a Game Loft for stories and other activities. There’s also a Treasure Chest where children can save things they’ve created or discovered.
The activities are deliberately limited in scope. A conversation in the Question Room ends after eight replies. Stories have six scenes. Trivia games contain ten questions. There are no streaks, endless feeds, or mechanisms designed to keep the child engaged indefinitely. I wanted using Marmalade to feel more like sitting down to do something, finishing it, and moving on, rather than opening an application that always has another reason for you to stay.
The Story Relay activity grew directly out of what I’d observed with my daughter’s fan fiction. Instead of generating an entire story in response to a short instruction, Marmalade encourages the child to participate in developing the narrative. The story progresses in stages, with the child deciding what happens next, contributing ideas, and making choices about characters and events. The AI can suggest possibilities and offer writing tips, but the experience is structured around the child’s participation.
The point isn’t to produce the most polished story possible. It’s to help her become a better storyteller.
The Study Nook follows a similar principle. If a child needs help with an assignment, the application is supposed to guide them through the problem, ask questions, explain difficult concepts, and help them organize their thoughts. It isn’t intended to produce finished homework for submission. I’m interested in how far this approach can go, particularly as children become increasingly accustomed to AI systems that will happily complete almost any task placed before them.
The Field Guide addresses the other incident, the frustrating afternoon spent trying to research a school presentation. It uses web search to gather information and presents explanations alongside sources that the child can examine. I’ve been particularly keen on encouraging the habit of checking where information comes from. AI makes it remarkably easy to obtain answers, but I worry that it also makes it easy to forget that answers need to be evaluated. If my daughter learns to treat an AI response as a starting point for investigation rather than the final word on a subject, I’ll consider that progress.
I don’t know yet how effective all of these approaches will prove to be. Marmalade is still a young project, and much of what I’ve built reflects my own intuitions about learning rather than the results of controlled educational research. But having a working application means I can experiment, observe what happens, and change things when they don’t work.
And my daughter has opinions of her own, which brings me to the cat.
Marmalade, the cat
At some point during development, my daughter decided that the application needed a cat. I added a small animated ginger cat that could be petted, would purr, curl up, play with yarn, and generally behave in ways that seemed appropriate for a cat living in a digital treehouse.
The cat acquired the name Marmalade, which eventually became the name of the entire project.
I had spent considerable time thinking about privacy boundaries, usage limits, educational activities, and the architecture needed to support different AI providers. She wanted a cat. In retrospect, this was a useful reminder that a child’s experience of software is not necessarily shaped by the things its developer considers most important.
I built the guardrails. She found the good parts.
The treehouse and the cat make the application feel less like a productivity tool and more like a place she can visit. I hadn’t fully appreciated how much that would matter. The design of educational software often begins with what adults want children to accomplish. But children still have to want to use it, and enjoyment needn’t be at odds with learning.
Why I made it open source
I built Marmalade using AI coding tools, primarily Gemini through Antigravity, with Claude Code helping me review and audit the code, particularly around privacy and safety. I’m not a full-time software engineer, and these tools allowed me to develop something considerably more ambitious than I would otherwise have attempted on my own.
There’s an apparent contradiction here, of course. I’ve used AI extensively to build an application intended to prevent a child from relying too heavily on AI. But I don’t really see it as a contradiction. Sometimes having a machine do the work is exactly what we want. At other times, doing the work ourselves is the point. I think that distinction matters enormously in education, and we’re only beginning to work through its implications.
Marmalade started as a personal project for my family. My daughter uses our own private instance at home, and I intend to keep improving it based on her experience. But as I worked on it, I began to think that other parents might be wrestling with some of the same questions. They might have different views about appropriate boundaries or which AI models their children should use, and that’s perfectly reasonable.
So I’ve released Marmalade as an open-source project under the MIT licence. Anyone can download it, examine the source code, and run their own instance on a computer, Raspberry Pi, or private server. It supports local AI models through Ollama, as well as supported commercial AI providers, subject to their terms and the relevant privacy considerations. I’m not planning to operate it as a commercial service.
It’s still early. The current version is designed for one child per household, there are features I haven’t implemented, and I’m certain there are things I’ve overlooked. I’m hoping that making the code public will encourage people to examine it critically, suggest improvements, and perhaps contribute to its development.
I don’t imagine that Marmalade resolves the larger questions around children and artificial intelligence. There are difficult issues involving privacy, dependency, educational development, and safety that no individual application can settle. I’m still trying to figure out what good AI use looks like for my own daughter, and I expect my answers will change as she grows older.
For now, though, she has a place where she can ask questions, research things, work on stories, and get help with schoolwork. I can set some boundaries without having to look over her shoulder. She gets to explore, and I get to worry a little less about how she’s doing it.
That’s a reasonable beginning.
If you’d like to try Marmalade, you can run it in Demo Mode without any AI API keys. The code and installation instructions are on GitHub. I’d be interested to hear what you’d change.