How to make a music game with AI
The frame clock and the audio clock are two different clocks, and the player only hears one of them. Time the whole game by that one.
Jump to a section
A rhythm game is judged entirely on timing, so the first thing to decide is which clock does the timing. Book every note ahead of time against the audio clock. Judge every key press against that same clock. Get that right and the rest of the game is ordinary work.
An animation frame turns up roughly sixty times a second, and sometimes it does not turn up at all. The audio clock advances with the samples leaving the device and does not care what the page is doing. The player hears the audio clock. So the audio clock is the only one allowed to say when a beat happened.
Four steps to a game that stays in time#
One repeating drum pattern, scheduled properly, nothing else on screen. If that wanders, everything built on it wanders.
Print the raw gap in milliseconds on screen. You want to see the number, not a word like Good covering for it.
Late in the song is where frame driven timing gives itself away. The first bar always sounds fine, which is the trap.
A chart written while your own offset is wrong comes out wrong in the same direction, and you will not hear it.
Build a four lane rhythm game as ONE self-contained .html file. No
frameworks and no audio files. Every sound is made with Web Audio
oscillators and filtered noise, so the page carries its own music.
AUDIO CLOCK: create one AudioContext. Every time in this game is a time
on its clock. Nothing is ever timed by counting frames.
START: the context begins suspended, so show a START button and call
resume() from inside that click. Wait until the context reports running,
then set the song start a short way in the future.
CHART: build the note list in code from a tempo in BPM and a repeating
four bar pattern across the lanes. A note holds a lane and a time in
seconds measured from the song start.
SCHEDULING: run a lookahead on a plain timer. Every 25 milliseconds,
find every note and drum hit beginning within the next 0.15 seconds and
book it with an exact start time. Never start a sound at the moment you
notice it is due.
VISUALS: each frame, read the audio clock once and work out where every
note belongs from that one number. The notes follow the audio. The audio
never follows the notes.
INPUT: D F J K hit the four lanes. On a key down, read the audio clock
and subtract the time of the nearest unhit note in that lane. Under
0.045 is PERFECT, under 0.09 is GOOD, under 0.15 still counts, past that
it is a miss. Say EARLY or LATE as well as the grade. Ignore the
browser's key repeat.
CALIBRATION: a screen that loops a bare click and asks the player to tap
along for sixteen beats. Take the median difference, keep it in
localStorage, and shift every later judgement by it. Let it be typed in
by hand too.
ELSEWHERE: anything that is not a note, such as a background pulse, moves
by the seconds elapsed since the last frame, clamped to 0.05.
PHONE: four tap zones across the bottom, and
canvas { touch-action: none; } in the CSS.
CONFIG: one object at the top holding BPM, the three judgement windows,
the lookahead, the scroll speed and the lane keys, each commented.The scheduling block and the input block are the game. Change the tempo, the pattern and the keys as much as you like. Leave the lookahead and the audio clock reads where they are, because a later request for a visual effect is exactly what quietly replaces them with a frame counter.
The frame clock and the audio clock#
Two timers run in a browser tab and they do not agree. An animation frame hands you a time from the page's own clock, and it arrives when the compositor is ready for it. The audio context reports a time that advances with the sound actually reaching the speaker.
Play a note because a frame told you to and the frame's error becomes the beat's error. One frame of lateness is nothing. The problem is that the errors do not cancel out. Each beat is placed relative to the one before it, so the gap accumulates for the length of the song. By the end the drums and the falling notes are plainly apart.
Scheduling ahead fixes it because a note's time is worked out from the song start, never from the previous note. Beat forty is the start time plus forty times the seconds per beat. That expression cannot drift. A lookahead loop on a plain timer peers a fraction of a second into the future and books anything due with an exact start time.
The same rule runs the other way for drawing. Each frame, read the audio clock once and derive every note position from that number. Do not nudge the notes along by the elapsed time. When the tab stutters, the notes should jump to wherever the music has got to, because the music is the thing the player is actually following.
Judging a hit without counting frames#
A judgement is one subtraction. Take the audio time at the moment the key went down, take away the note's scheduled time, and look at how big the remainder is. Its sign tells you early or late. Counting frames since the note appeared gives a different answer on a slow machine, which makes the score a report on somebody's hardware.
Three windows are enough, and all three belong in the config object. One tight, for a clean hit. One generous, for a scrappy one. A third that still scores rather than failing, so a player slightly out of time is not punished as though they had done nothing. Beyond the last window the note is missed and must stay unhit.
- Say early or late, not just the grade. A player told Good learns nothing. A player told Late twice running moves their hands, which is the whole feedback loop of the genre.
- One press, one note. Mark a note hit and never let a second press claim it. Key repeat from a held key has to be ignored, or leaning on a key farms a lane.
- Judge the nearest unhit note in that lane. Not the first one in the list, which may already be behind the player, and not the nearest in any lane.
- Keep missed notes in the list. The score needs them, and so does the player, who should be able to see what sailed past.
Why nothing plays until the player taps#
Browsers will not let a page make noise by itself. An audio context created before any interaction begins suspended, and it stays suspended until something resumes it from inside a real click or tap. A rhythm game written without that in mind loads to a silent screen with notes already falling down it.
So a start button is not politeness in this genre. It is the thing that starts the clock. Put the resume call in the click handler, wait for the context to report that it is running, then set the song start a little way into the future. Starting at exactly the current audio time makes the first note late before anybody has touched a key.
That button is also the right place for the setup the player should never notice. Build the oscillators, work out the chart, read the saved calibration offset. A moment of work behind a button is invisible. The same work during the first bar is a stutter, and the player will assume their machine is at fault.
Calibration, and why every music game has one#
Two delays sit between your code and the player's ear, and neither of them is yours. The operating system buffers audio before it leaves the device. Wireless headphones buffer it again, by an amount that is easy to feel. A player on wireless is late on every note, scores badly, and decides the game is unfair.
A calibration screen turns all of that into a single number. Loop a bare click, ask the player to tap along for a few bars, and take the median difference between their taps and the beats. Median rather than mean, because one distracted tap would drag an average a long way for no reason.
| The player says | What it usually is | Ask for |
|---|---|---|
| It drifts out by the end | Each beat timed from the previous beat | Every note time derived from one song start |
| Everything feels late | No calibration offset anywhere | A calibration screen and a stored offset |
| It stutters on my laptop | Setup work happening during the song | All of it moved behind the start button |
| The notes jump about | Note positions advanced by frame delta | Positions derived from the audio clock each frame |
| Nothing happens at all | The context is still suspended | resume() inside the click, then wait for running |
Report the symptom rather than guessing the line. Four of these five are the same mistake wearing different clothes.
🎵Music and rhythm games written by AI
Gemini 3.8 FlashCommon questions#
Can AI make a rhythm game?
Yes, and it can write the music too, as oscillators and noise rather than a file. The part to be careful about is timing. A model asked for a rhythm game without any mention of scheduling will often reach for the animation frame, which is the one approach that cannot hold time.
Why does my rhythm game drift out of time?
Each beat is being timed from the last one, so every small error is kept and added to. Work out each note time from a single song start instead, as the start plus the beat number times the seconds per beat. That expression has nothing to accumulate.
Why is there no sound until I click something?
An audio context created on page load starts suspended, and only a real click or tap can resume it. This is deliberate browser behaviour and there is no way round it. Use it: make the start button the thing that resumes the context and sets the song start time.
How do I judge whether a hit was on time?
Read the audio clock inside the input handler and subtract the scheduled time of the nearest unhit note in that lane. Grade the size of the difference and report its sign as early or late. Never count frames between the note appearing and the key going down.
Do I need music files for a music game?
No. A drum kit and a bass line are a few oscillators, a noise burst and a filter, which keeps the whole game in one HTML file with nothing to host. It also means the tempo is a number you can change rather than a property of a recording.
Players say the timing feels late. What do I change?
Not the windows. Add a calibration screen and store the offset, because the delay is coming from their headphones or their output device, not from your judgements. Widening the windows for everyone hides the problem and makes the game easier for people who never had it.
Where to go next#
A rhythm game is worth sharing the moment one pattern feels good, because a short chart played twice beats a long one played once. Put it up and it gets a page with the model and the prompt on it, and a leaderboard, which this genre happens to suit better than most.
The general approach is in how to make a game with AI. Timing windows and input feel carry over almost whole from the reflex game guide, and the drawing side is in the 2D guide. Prompts for particular faults live in the prompt guide, and the music games here are worth playing with headphones on before you write your own.
Built something? Put it on Agent Games.
Publishing takes a file and a minute. Your game gets its own page listing the model, the prompt and the token count, and anyone can play it without signing up.
Read next