Tutorly connects students with tutors through courses, live classrooms, 1:1 voice and video calls, an AI tutor, quizzes, study plans, Shorts, podcasts, scientific articles, and communities. It is built for multi-country curricula (country → education system → grade level → institution), with Zambia as the launch market and purchasing-power pricing.
I designed and built the whole system: an Expo app for students on iOS, Android, and web; a Tauri desktop studio where tutors teach, record, and edit; a Bun + Hono server exposing GraphQL as its primary API; and a Rust worker that turns uploads into streamable, data-friendly media.
Much of the engineering is shaped by the market — mobile data is expensive, devices are modest, many learners are minors, and many study above the official age for their grade — so the platform is built to be cheap to stream, safe by default, and welcoming rather than gatekeeping.
AI Tutor
- Gemini-powered tutor that streams answers over SSE and accepts text, images, documents, and voice notes.
- Real-time voice tutoring: the server bridges the app’s WebSocket to a Gemini Live session, so the API key never reaches the client and guardrails, metering, and safeguarding still apply.
- Tool calling with 19 tools — search the curriculum, build a practice quiz, grade work, generate a diagram, review past mistakes, save notes, book a live session, enrol in a course, set a study reminder, and more.
- Tool schemas never accept a user ID; the server binds the authenticated student itself, closing the most common IDOR hole in tool-calling apps.
- Answers are grounded in published courses via a pgvector RAG index (HNSW) and in lesson transcripts, so the tutor can answer from what a teacher actually said.
- Three-layer guardrails: a cheap Flash-Lite pre-classifier, a constrained main call, and an output check — including detection of requests to do graded work.
- Safeguarding: if a child signals self-harm, the tutor keeps talking supportively while a durable escalation reaches a human, with regional crisis resources.
- Cost control that makes “unlimited AI” safe on a K99/month plan: rolling, tutoring-aware history compaction, prompt caching, daily caps, a per-period cost ceiling, and Gemini → Nova provider failover.
Live Classes, Calls & Radio
- Group classes and 1:1 sessions on self-hosted LiveKit, metered separately per plan because their costs differ.
- Session recording through LiveKit Egress, with replays for students who missed the class.
- Native phone-style calls: CallKit on iOS via VoIP push, FCM on Android, and the tutor’s desktop app rings over WebSockets.
- Live radio: audio-only broadcasts that automatically become podcast episodes — recorded, loudness-normalised, streamed, and published, with the tutor notified.
- Session requests, tutor availability, and scheduled reminders.
Rust Media Pipeline
- A standalone Rust worker driven by a pg-boss queue and Postgres NOTIFY, running FFmpeg and writing results back to object storage.
- Adaptive HLS from 1080p down to an always-present 360p rung for mobile data — never upscaled, portrait stays portrait.
- Listen-only audio for learners on data bundles and compressed offline downloads for paid plans.
- whisper.cpp transcription produces captions, word-level timings, and searchable transcripts.
- Signed HLS playback out of a private bucket, with the signature in the path so native players that can’t send headers still work.
- Speaker tracking for vertical Shorts: a ~1 MB UltraFace ONNX model compiled into the binary finds the tutor’s face and builds a smooth “patient camera operator” crop path from landscape lessons.
- Audio processing for podcasts: loudness levelling, silence detection, and waveforms.
Tutor Desktop Studio
- Tauri 2 app for tutors: teach live, author courses and quizzes, manage students, track earnings, and moderate communities.
- Non-destructive video editor: the timeline is a JSON document previewed live and rendered server-side; text layers are rasterised by the app so the render matches the preview exactly.
- Edit a video by editing its transcript — deleting words cuts the matching source ranges, as a single undo step.
- An AI editing assistant proposes cuts that are applied against the current edit, even if it changed since the proposal.
- Shorts builder: pick moments from a lesson and get 9:16 clips with 2–3 word captions sized for phones.
- Podcast audio editor: trim, cut pauses and “um”s, add intro/outro and a music bed, fade, and level loudness — the original is always restorable, and chapters and listeners’ positions move with the edit.
- Brand kits (logo, colours, font, intro/outro), including automatic removal of a logo’s solid background with an edge flood fill.
- Crash-safe recording saved to IndexedDB every second, access tokens in the OS keychain, and signed auto-updates.
- Student insights: study time, course progress, and quiz performance per learner.
Privacy & Safety
- End-to-end encrypted chat with NaCl keys: the server stores only public keys and opaque message envelopes.
- PIN-protected key recovery using scrypt, with server-enforced brute-force lockout.
- Rich messages — equations, code, questions, assignments — that keep everything the user wrote inside the encrypted body.
- Guardian consent for minors, and a “guide, don’t gate” age policy so over-age learners (common in Zambia) are never turned away.
- Blocking, reporting, and moderation queues across chat, communities, and comments.
Learning & Content
- Courses with video, interactive, and text lessons, bookmarks, notes, and comments.
- Quizzes with feedback, AI-generated practice quizzes, study plans, progress tracking, and streaks.
- Shorts, podcasts with scheduled episodes and subscriptions, and topic communities with events.
- Scientific articles with a block editor, co-authors, moderation, and PubMed/PMC import via NCBI E-utilities, rate-limited to stay within NCBI policy.
- Offline-first mobile app with a persisted Apollo cache and downloadable lessons, plus a 3D onboarding built with React Three Fiber.
Business & Growth
- Purchasing-power pricing (K99 locally vs $19 in USD markets), with trials, entitlements frozen at purchase, and allowances that reset on the billing anniversary.
- New markets are a data change, not a deploy — currencies for South Africa, Botswana, and Malawi are already seeded.
- WhatsApp support bot that links a number to an account, answers from that account, books or enrols only after a confirm tap, and hands schoolwork to the in-app tutor.
- Notifications over Expo push, email, and SMS (eSMS Africa), with per-user preferences.
- Object storage on S3 with automatic Cloudflare R2 failover and an optional mirror mode.
Architecture
- Layered routes/resolvers → controllers → services → Prisma, so GraphQL and REST share the same business logic.
- About 5,000 lines of GraphQL schema across 29 domains; typed operations on mobile with gql.tada and live updates on desktop with graphql-ws.
- Native Bun WebSockets for chat, calls, live sessions, and voice tutoring, with Redis pub/sub.
- Local JWT plus Auth0 with just-in-time user provisioning.
- Deployed on Fly.io and an Oracle VPS, with LiveKit, Egress, and the media worker in containers.