- 6×throughput increase
- -58%unit transcoding cost
- 99.95%job success rate
- < 8minavg. for a 1-hour video
Background
The client runs a content platform where tens of thousands of user-uploaded videos need transcoding into multiple renditions and packaging for delivery each day. The original setup was FFmpeg scripts on one large bare-metal box with a database-polling queue: at peak the backlog reached over ten hours, a machine failure meant losing the whole queue, and operations staff were manually re-running jobs.
The hard parts
- Long jobs can’t restart from zero. A two-hour 4K video takes tens of minutes; failing halfway and starting over is unaffordable in both cost and latency.
- Wildly uneven load. Daytime upload peaks exceed overnight idle by more than tenfold — fixed capacity is either wasted or insufficient.
- Cost sensitivity. Transcoding is pure CPU/GPU work and scales almost linearly, so the encoding parameters themselves had to be addressed.
- Mixed priorities. Paying customers’ videos must jump the queue; bulk re-encoding of the back catalogue must not delay new uploads.
What we built
Sharding and resumable jobs
Each video is split at GOP boundaries into independent shards, scheduled per shard, retried per shard on failure, and losslessly recombined with concat. The cost of a failure drops from “redo the whole video” to “redo one shard.”
Elastic cluster
- Scheduler written in Go, job state in Redis, event stream over Kafka
- Workers run as Kubernetes Jobs, autoscaled on queue depth
- Scales down to a minimum overnight, up to hundreds of workers within minutes at peak
- Spot instances handle low-priority work; reclaimed shards simply re-enter the queue
Multi-level priority queues
Three tiers — paid realtime > standard upload > bulk re-encode — with preemptive scheduling: a high-priority job can pause a low-priority shard, protecting the latency guarantee for VIP content.
Encoding parameter tuning
x264/x265 parameters and filter chains were tuned separately per content type (lecture screen capture, live action, animation), with VMAF-based quality scoring for automated comparison. Average bitrate fell more than 30% at unchanged perceived quality, saving both bandwidth and storage.
Output and delivery
Uniform output of multi-rendition HLS (360p/720p/1080p/original) plus thumbnail sprites and keyframe posters, written to object storage and pre-warmed on the CDN for adaptive playback.
Outcomes
- Throughput rose from roughly 1,200 to over 7,500 videos per day; peak backlog disappeared
- Unit cost down 58% (elastic scaling, spot instances and encoding tuning combined)
- Job success rate up from 96.2% to 99.95%, with manual intervention essentially eliminated
- A full operational dashboard: queue depth, worker utilization, failure breakdown, per-job cost
What it became
The scheduling framework was abstracted into an internal component that similar projects reuse directly, cutting their delivery time roughly in half.
This pipeline also became the carrier for AI capability: because decoding is already happening, frame sampling and audio extraction hang off it as a side branch, so adding content understanding and moderation raised unit processing cost by only 0.9% — see AI Video Understanding & Moderation.
Related services: Software Development and Technical Support & Operations.