HyperFrames, which allows you to create videos using code, seems to be amazing, so I tried it out.
HyperFrames, which allows you to create videos using code, seems to be amazing, so I tried it out.
When I tried HyperFrames, a framework for creating videos using AI, I was able to create MP4 files just by writing HTML. We will organize the records and how they differ from Remotion from an AI agent's perspective.
The trigger was the word “It seems amazing”
It all started when I happened to come across the name HyperFrames while doing some research. It says ``You can create MP4 from HTML'' and ``It's compatible with AI agents.'' <br>Honestly, I didn't really believe it at first. All video generation tools claim to be ``easy,'' but when you actually use them, they often require a lot of settings or have problems with the output quality. <br>However, I can't talk about it unless I try it, so I actually made a 30-second explanation video. Judging from the results, the MP4 came out properly. And sooner than I expected.
How to output MP4 just by writing attributes in HTML
HyperFrames is an open source framework published by HeyGen that allows you to assemble videos by adding timing attributes to HTML.
When you arrange elements with the attributes `data-start` (starting seconds), `data-duration` (length), and `data-track-index` (layer number) in HTML, HyperFrames takes a screenshot one frame at a time using headless Chrome, synthesizes it with audio using FFmpeg, and converts it to MP4.
<div class="clip" data-start="0" data-duration="10" data-track-index="1">
Contents of scene 1
</div><audio class="clip" data-start="0" data-duration="30" data-track-index="8" src="bgm.wav" data-volume="0.3"> </audio> ````
Only elements with `class="clip"` are subject to time control. If you forget to add it, the element will not be hidden, so you need to be careful, but other than that, it can be written almost as an extension of normal HTML production. <br>Knowing this mechanism will help you quickly troubleshoot when things are not working properly. Capturing screenshots with headless Chrome and synthesizing audio with FFmpeg are separate processes, so what you should look at will depend on whether ``the image is correct but there is no sound'' or ``the image itself is not what you intended.''
What I actually made
I tried porting the components of an explanation video that I had already created using Remotion to HyperFrames. If you want to measure the power of a new framework, it's easier to compare it by porting an existing one than building one from scratch.
The composition is as follows.
- 3 scenes x 10 seconds = 30 seconds - Generate Japanese TTS audio with VOICEVOX - Generate lo-fi ambient BGM with ACE-Step (about 30 seconds) - Scene transition animation with GSAP - Update clock and progress bar every frame with `hf-seek` event
Audio placement can also be written using HTML attributes. The background music was played for 30 seconds, and the audio for each scene was delayed a little from the start of the scene. When the narration starts immediately after a scene change, there is a pause and it is difficult to hear, so it would have sounded more natural if there was a slight pause. <br>When I lowered the BGM volume with `data-volume`, the BGM receded into the background in the rendered audio, and TTS came to the forefront. This is an officially supported attribute for HyperFrames, and just passing a value between 0 and 1 will work. <br>When I executed `npm run render`, an MP4 was exported in a few minutes.
What I'm addicted to
I'll write about some of the issues I encountered during the porting process.
GSAP and CSS transforms collide
If `transform: translate(-50%, -60%)` is specified in CSS for an element animated with GSAP, the GSAP tween and CSS transform specification may conflict and the element may not move to the intended position. Both handle the same `transform` property, so if you manage the base position of translate and the amount of movement of the animation separately, it will be difficult to follow the appearance. <br>The solution was to delete the CSS transform and rewrite it as `xPercent` / `yPercent` in GSAP's `fromTo`. It is easier to avoid confusion if you make a rule that the transformation of the element you want to move with GSAP is managed only with GSAP.
tl.fromTo("#title",
{ opacity: 0, xPercent: -50, yPercent: -60 },
{
opacity: 1,
xPercent: -50,
yPercent: -60,
duration: 1.2,
ease: "power3.out"
},
0
);
An error will occur if track-index is duplicated
HyperFrames will issue an `overlapping_clips_same_track` error if elements with the same `data-track-index` number exist in the same time period. As the number of HTML elements increases, it is easy to overlook them. <br>It is necessary to assign a unique number to each element such as backdrop, grid, header, main card, presenter, subtitle, footer, background music, and audio.
Elements do not disappear if you forget `class="clip"`
`class="clip"` is required for elements that you want to be subject to time control, but even if you forget to add it, it will be difficult to notice because it will be displayed as static HTML. If you feel that there is an element that remains visible after rendering, check this first.
Japanese TTS uses VOICEVOX
HyperFrames has a built-in TTS called Kokoro-82M. When I tried it, I found that the Japanese accent was more English-like, making it unsuitable for Japanese explanation videos. <br>When I switched to VOICEVOX, which runs locally, I was able to generate natural Japanese voices. The integration itself was simple, with two steps: generating a voice query using the REST API and then synthesizing it.
BGM was generated by ACE-Step
I wanted to create my own BGM, so I tried ACE-Step, an open source music generation model. <br>In the prompt, genre and mood are separated by commas. It was specified by combining keywords such as electronic, ambient, lo-fi, chill, instrumental, soft beat, calm, and minimal. <br>It runs on Apple Silicon (MPS) and outputs a WAV file of several tens of seconds. I had to be careful that Torchaudio would issue a warning if the output format was not WAV, and the file would be an odd size.
Compare Remotion and HyperFrames
Through porting, we could clearly see the difference in their ``specialized granularity.''
HyperFrames: Quickly shape, easy to pass to agents
Since it is HTML-based, it is easy for AI agents to read and write code. If you write `data-start="10"`, you can see at a glance that it starts from 10 seconds. CSS and GSAP animations can be written using web knowledge, making it easy for agents to use their existing knowledge. <br>Suitable for quickly turning rough videos into shapes. It is suitable for purposes in which HTML, CSS, audio, BGM, and scene composition are passed to the AI all at once, and then the MP4 data is retrieved.
Remotion: Pack by frame, reuse as parts
Remotion can handle time in frames. You can specify the starting frame and number of frames to display with `Sequence`, take the current frame with `useCurrentFrame`, and add physically-based movement with `spring`. <br>High degree of freedom in voice control. Since volume, trimming, playback speed, and mute can be handled as codes, ducking, which lowers the background music only during narration, can be naturally implemented.
const frame = useCurrentFrame();
const volume = interpolate(
frame,
[0, 30, 60, 90],
[0.2, 0.05, 0.05, 0.2],
{ extrapolateLeft: 'clamp', extrapolateRight: 'clamp' }
);
<Audio src={bgmSrc} volume={volume} />
In addition, reused effects can be made into React components. Remotion is easier to manage if you want to reuse the same structure for multiple videos, or if you want to output subtitles by word.
How to use them properly
Broadly speaking, HyperFrames is suitable for applications where an AI agent is required to assemble a short video, or where the initial form is generated quickly. <br>On the other hand, Remotion is easier to handle for precise adjustment on a frame-by-frame basis, subtitles on a word-by-word basis, ducking of background music and audio, and reuse of production as a React component.
Dynamically update with `hf-seek` event
HyperFrames fires the `hf-seek` event every time a frame advances. This event allows you to dynamically update DOM elements based on frame numbers. <br>In this video, we used this event to control the clock display, progress bar, code highlight line, and mini avatar animation.
window.addEventListener("hf-seek", (e) => {
const frame = e.detail.frame;
const sec = frame / 30; // 30fps// update clock document.getElementById("clock").textContent = fmt(sec);
// Update progress bar document.getElementById("progress").style.width = (sec / duration * 100) + "%"; }); ````
Non-deterministic logic such as `Math.random()` and `Date.now()` cannot be used, but calculations using frame numbers are deterministic, so there is no problem.
What happens when you leave it to an AI agent?
The porting this time was left to an AI agent. I asked them to reproduce Remotion's components as they are in HyperFrames. <br>The Remotion code has layout, animation, audio, and timing all mixed together in React components, so converting to another framework is a fairly complicated task. <br>During the process, I encountered a duplicate track-index error and a transform conflict between GSAP and CSS, but the error message was easy to understand, and I pasted the error and asked for a fix, and it was resolved in a short exchange. I think the reason is the HTML attribute design. <br>If you write `data-start="10"`, it will say "starting from 10 seconds". If you write `data-duration="10"`, it will say "displayed for 10 seconds". This is a format suitable for agents to read and write, and the intent of ``this scene is from how many seconds to how long'' is directly expressed in the code. <br>Remotion is often written in frames assuming 30fps, such as `<Sequence from={300} durationInFrames={300}>`, which requires some effort to convert to seconds. For agents, HyperFrames was easier to handle in many situations.
Thoughts after trying it
I started out saying, ``It seems amazing,'' and within a few hours, a working MP4 was released. <br>Honestly, I was skeptical until it started working. I had the impression that the more ``easy'' a video generation tool is, the more difficult it is to take the final step. However, the HTML attribute design was straightforward, and the modification cycle when passed to the AI agent was short. <br>I still think that Remotion is better for precise control. However, HyperFrames is faster at the task of "creating the initial shape." If you want AI to create videos, using HyperFrames as the entrance is a very realistic option.
Reference
- [HyperFrames Data Attributes](https://hyperframes.heygen.com/concepts/data-attributes) - [Remotion documentation](https://www.remotion.dev/docs) - [VOICEVOX](https://voicevox.hiroshiba.jp/) - [ACE-Step](https://github.com/ace-step/ACE-Step)
---
*This post was machine-translated (Google Translate) from a Japanese original at [nomuraya-hub.pages.dev](https://nomuraya-hub.pages.dev/). Pre-review draft. I am the same author writing under different pen names — "nomuraya / shimajima / 中翔" — depending on the medium.*