rob
/
moq-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #15 from kixelated/quiche 

Switch from Go to Rust
This commit is contained in:
kixelated 2023-05-22 15:25:06 -07:00 committed by GitHub
parent 7bb6068951 4585a0c04e
commit 5410a3767f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
81 changed files with 3227 additions and 2280 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

69
README.md
View File

@ -1,64 +1,39 @@
# Warp
Segmented live media delivery protocol utilizing QUIC streams. See the [Warp draft](https://datatracker.ietf.org/doc/draft-lcurley-warp/).
Live media delivery protocol utilizing QUIC streams. See the [Warp draft](https://datatracker.ietf.org/doc/draft-lcurley-warp/).
Warp works by delivering each audio and video segment as a separate QUIC stream. These streams are assigned a priority such that old video will arrive last and can be dropped. This avoids buffering in many cases, offering the viewer a potentially better experience.
Warp works by delivering media over independent QUIC stream. These streams are assigned a priority such that old video will arrive last and can be dropped. This avoids buffering in many cases, offering the viewer a potentially better experience.
# Limitations
## Browser Support
This demo currently only works on Chrome for two reasons:
This demo requires WebTransport and WebCodecs, which currently (May 2023) only works on Chrome.
1. WebTransport support.
2. [Media underflow behavior](https://github.com/whatwg/html/issues/6359).
# Development
## Easy Mode
Requires Docker *only*.
The ability to skip video abuses the fact that Chrome can play audio without video for up to 3 seconds (hardcoded!) when using MSE. It is possible to use something like WebCodecs instead... but that's still Chrome only at the moment.
```
docker-compose up --build
```
## Streaming
This demo works by reading pre-encoded media and sleeping based on media timestamps. Obviously this is not a live stream; you should plug in your own encoder or source.
Then open [https://localhost:4444/](https://localhost:4444) in a browser. You'll have to click past the TLS error, but that's the price you pay for being lazy. Follow the more in-depth instructions if you want a better development experience.
The media is encoded on disk as a LL-DASH playlist. There's a crude parser and I haven't used DASH before so don't expect it to work with arbitrary inputs.
## QUIC Implementation
This demo uses a fork of [quic-go](https://github.com/lucas-clemente/quic-go). There are two critical features missing upstream:
1. ~~[WebTransport](https://github.com/lucas-clemente/quic-go/issues/3191)~~
2. [Prioritization](https://github.com/lucas-clemente/quic-go/pull/3442)
## Congestion Control
This demo uses a single rendition. A production implementation will want to:
1. Change the rendition bitrate to match the estimated bitrate.
2. Switch renditions at segment boundaries based on the estimated bitrate.
3. or both!
Also, quic-go ships with the default New Reno congestion control. Something like [BBRv2](https://github.com/lucas-clemente/quic-go/issues/341) will work much better for live video as it limits RTT growth.
# Setup
## Requirements
* Go
* Rust
* ffmpeg
* openssl
* Chrome Canary
* Chrome
## Media
This demo simulates a live stream by reading a file from disk and sleeping based on media timestamps. Obviously you should hook this up to a real live stream to do anything useful.
Download your favorite media file:
Download your favorite media file and convert it to fragmented MP4:
```
wget http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4 -O media/source.mp4
./media/fragment
```
Use ffmpeg to create a LL-DASH playlist. This creates a segment every 2s and MP4 fragment every 10ms.
```
./media/generate
```
You can increase the `frag_duration` (microseconds) to slightly reduce the file size in exchange for higher latency.
## TLS
## Certificates
Unfortunately, QUIC mandates TLS and makes local development difficult.
If you have a valid certificate you can use it instead of self-signing. The go binaries take a `-tls-cert` and `-tls-key` argument. Skip the remaining steps in this section and use your hostname instead.
If you have a valid certificate you can use it instead of self-signing.
Otherwise, we use [mkcert](https://github.com/FiloSottile/mkcert) to install a self-signed CA:
```
@ -72,20 +47,18 @@ The Warp server supports WebTransport, pushing media over streams once a connect
```
cd server
go run main.go
cargo run
```
This can be accessed via WebTransport on `https://localhost:4443` by default.
This listens for WebTransport connections (not HTTP) on `https://localhost:4443` by default.
## Web Player
The web assets need to be hosted with a HTTPS server. If you're using a self-signed certificate, you may need to ignore the security warning in Chrome (Advanced -> proceed to localhost).
## Web
The web assets need to be hosted with a HTTPS server.
```
cd player
cd web
yarn install
yarn serve
```
These can be accessed on `https://localhost:4444` by default.
If you use a custom domain for the Warp server, make sure to override the server URL with the `url` query string parameter, e.g. `https://localhost:4444/?url=https://warp.demo`.

3
cert/.dockerignore Normal file
View File

@ -0,0 +1,3 @@
*.crt
*.key
*.hex

1
cert/.gitignore vendored
View File

@ -1,2 +1,3 @@
*.crt
*.key
*.hex

22
cert/Dockerfile Normal file
View File

@ -0,0 +1,22 @@
# Use ubuntu because it's ez
FROM ubuntu:latest
WORKDIR /build
# Use openssl and golang to generate certificates
RUN apt-get update && \
apt-get install -y ca-certificates openssl golang xxd
# Download the go modules
COPY go.mod go.sum ./
RUN go mod download
# Copy over the remaining files.
COPY . .
# Save the certificates to a volume
VOLUME /cert
# TODO support an output directory
CMD ./generate && cp localhost.* /cert

2
cert/generate
View File

@ -17,4 +17,4 @@ go run filippo.io/mkcert -ecdsa -install
go run filippo.io/mkcert -ecdsa -days 10 -cert-file "$CRT" -key-file "$KEY" localhost 127.0.0.1 ::1
# Compute the sha256 fingerprint of the certificate for WebTransport
openssl x509 -in "$CRT" -outform der | openssl dgst -sha256 > ../player/fingerprint.hex
openssl x509 -in "$CRT" -outform der | openssl dgst -sha256 -binary | xxd -p -c 256 > localhost.hex

45
docker-compose.yml Normal file
View File

@ -0,0 +1,45 @@
version: '3'
services:
# Generate certificates only valid for 14 days.
cert:
build: ./cert
volumes:
- cert:/cert
# Generate a fragmented MP4 file for testing.
media:
build: ./media
volumes:
- media:/media
# Serve the web code once we have certificates.
web:
build: ./web
ports:
- "4444:4444"
volumes:
- cert:/cert
depends_on:
cert:
condition: service_completed_successfully
# Run the server once we have certificates and media.
server:
build: ./server
environment:
- RUST_LOG=debug
ports:
- "4443:4443/udp"
volumes:
- cert:/cert
- media:/media
depends_on:
cert:
condition: service_completed_successfully
media:
condition: service_completed_successfully
volumes:
cert:
media:

1
media/.dockerignore Normal file
View File

@ -0,0 +1 @@
fragmented.mp4

25
media/Dockerfile Normal file
View File

@ -0,0 +1,25 @@
# Create a build image
FROM ubuntu:latest
# Create the working directory.
WORKDIR /build
# Install necessary packages
RUN apt-get update && \
apt-get install -y \
ca-certificates \
wget \
ffmpeg
# Download a file from the internet, in this case my boy big buck bunny
RUN wget http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4 -O source.mp4
# Copy an run a script to create a fragmented mp4 (more overhead, easier to split)
COPY fragment .
# Create a media volume
VOLUME /media
# Fragment the media
# TODO support an output directory
CMD ./fragment && cp fragmented.mp4 /media

12
media/fragment Executable file
View File

@ -0,0 +1,12 @@
#!/bin/bash
cd "$(dirname "$0")"
# empty_moov: Uses moof fragments instead of one giant moov/mdat pair.
# frag_every_frame: Creates a moof for each frame.
# separate_moof: Splits audio and video into separate moof flags.
# omit_tfhd_offset: Removes absolute byte offsets so we can fragment.
ffmpeg -i source.mp4 -y \
-c copy \
-movflags empty_moov+frag_every_frame+separate_moof+omit_tfhd_offset \
fragmented.mp4 2>&1

18
media/generate
View File

@ -1,18 +0,0 @@
#!/bin/bash
ffmpeg -i source.mp4 \
-f dash -ldash 1 \
-c:v libx264 \
-preset veryfast -tune zerolatency \
-c:a aac \
-b:a 128k -ac 2 -ar 44100 \
-map v:0 -s:v:0 1280x720 -b:v:0 3M \
-map v:0 -s:v:1 854x480 -b:v:1 1.1M \
-map v:0 -s:v:2 640x360 -b:v:2 365k \
-map 0:a \
-force_key_frames "expr:gte(t,n_forced*2)" \
-sc_threshold 0 \
-streaming 1 \
-use_timeline 0 \
-seg_duration 2 -frag_duration 0.01 \
-frag_type duration \
playlist.mpd

121
player/src/audio/decoder.ts
View File

@ -1,121 +0,0 @@
import * as Message from "./message";
import * as MP4 from "../mp4"
import * as Stream from "../stream"
import * as Util from "../util"
import Renderer from "./renderer"
export default class Decoder {
// Store the init message for each track
tracks: Map<string, Util.Deferred<Message.Init>>;
decoder: AudioDecoder; // TODO one per track
sync: Message.Sync;
constructor(config: Message.Config, renderer: Renderer) {
this.tracks = new Map();
this.decoder = new AudioDecoder({
output: renderer.emit.bind(renderer),
error: console.warn,
});
}
init(msg: Message.Init) {
let defer = this.tracks.get(msg.track);
if (!defer) {
defer = new Util.Deferred()
this.tracks.set(msg.track, defer)
}
if (msg.info.audioTracks.length != 1 || msg.info.videoTracks.length != 0) {
throw new Error("Expected a single audio track")
}
const track = msg.info.audioTracks[0]
const audio = track.audio
defer.resolve(msg)
}
async decode(msg: Message.Segment) {
let track = this.tracks.get(msg.track);
if (!track) {
track = new Util.Deferred()
this.tracks.set(msg.track, track)
}
// Wait for the init segment to be fully received and parsed
const init = await track.promise;
const audio = init.info.audioTracks[0]
if (this.decoder.state == "unconfigured") {
this.decoder.configure({
codec: audio.codec,
numberOfChannels: audio.audio.channel_count,
sampleRate: audio.audio.sample_rate,
})
}
const input = MP4.New();
input.onSamples = (id: number, user: any, samples: MP4.Sample[]) => {
for (let sample of samples) {
// Convert to microseconds
const timestamp = 1000 * 1000 * sample.dts / sample.timescale
const duration = 1000 * 1000 * sample.duration / sample.timescale
// This assumes that timescale == sample rate
this.decoder.decode(new EncodedAudioChunk({
type: sample.is_sync ? "key" : "delta",
data: sample.data,
duration: duration,
timestamp: timestamp,
}))
}
}
input.onReady = (info: any) => {
input.setExtractionOptions(info.tracks[0].id, {}, { nbSamples: 1 });
input.start();
}
// MP4box requires us to reparse the init segment unfortunately
let offset = 0;
for (let raw of init.raw) {
raw.fileStart = offset
input.appendBuffer(raw)
}
const stream = new Stream.Reader(msg.reader, msg.buffer)
/* TODO I'm not actually sure why this code doesn't work; something trips up the MP4 parser
while (1) {
const data = await stream.read()
if (!data) break
input.appendBuffer(data)
input.flush()
}
*/
// One day I'll figure it out; until then read one top-level atom at a time
while (!await stream.done()) {
const raw = await stream.peek(4)
const size = new DataView(raw.buffer, raw.byteOffset, raw.byteLength).getUint32(0)
const atom = await stream.bytes(size)
// Make a copy of the atom because mp4box only accepts an ArrayBuffer unfortunately
let box = new Uint8Array(atom.byteLength);
box.set(atom)
// and for some reason we need to modify the underlying ArrayBuffer with offset
let buffer = box.buffer as MP4.ArrayBuffer
buffer.fileStart = offset
// Parse the data
offset = input.appendBuffer(buffer)
input.flush()
}
}
}

30
player/src/audio/message.ts
View File

@ -1,30 +0,0 @@
import * as MP4 from "../mp4"
import { RingInit } from "./ring"
export interface Config {
sampleRate: number;
ring: RingInit;
}
export interface Init {
track: string;
info: MP4.Info;
raw: MP4.ArrayBuffer[];
}
export interface Segment {
track: string;
buffer: Uint8Array; // unread buffered data
reader: ReadableStream; // unread unbuffered data
}
// Audio tells video when the given timestamp should be rendered.
export interface Sync {
origin: number;
clock: DOMHighResTimeStamp;
timestamp: number;
}
export interface Play {
timestamp?: number;
}

85
player/src/audio/renderer.ts
View File

@ -1,85 +0,0 @@
import * as Message from "./message"
import { Ring } from "./ring"
export default class Renderer {
ring: Ring;
queue: Array<AudioData>;
sync?: DOMHighResTimeStamp
running: number;
constructor(config: Message.Config) {
this.ring = new Ring(config.ring)
this.queue = [];
this.running = 0
}
emit(frame: AudioData) {
if (!this.sync) {
// Save the frame as the sync point
this.sync = 1000 * performance.now() - frame.timestamp
}
// Insert the frame into the queue sorted by timestamp.
if (this.queue.length > 0 && this.queue[this.queue.length-1].timestamp <= frame.timestamp) {
// Fast path because we normally append to the end.
this.queue.push(frame)
} else {
// Do a full binary search
let low = 0
let high = this.queue.length;
while (low < high) {
var mid = (low + high) >>> 1;
if (this.queue[mid].timestamp < frame.timestamp) low = mid + 1;
else high = mid;
}
this.queue.splice(low, 0, frame)
}
if (!this.running) {
// Wait for the next animation frame
this.running = self.requestAnimationFrame(this.render.bind(this))
}
}
render() {
// Determine the target timestamp.
const target = 1000 * performance.now() - this.sync!
// Check if we should skip some frames
while (this.queue.length) {
const next = this.queue[0]
if (next.timestamp >= target) {
break
}
console.warn("dropping audio")
this.queue.shift()
next.close()
}
// Push as many as we can to the ring buffer.
while (this.queue.length) {
let frame = this.queue[0]
let ok = this.ring.write(frame)
if (!ok) {
break
}
frame.close()
this.queue.shift()
}
if (this.queue.length) {
this.running = self.requestAnimationFrame(this.render.bind(this))
} else {
this.running = 0
}
}
play(play: Message.Play) {
this.ring.reset()
}
}

146
player/src/audio/ring.ts
View File

@ -1,146 +0,0 @@
// Ring buffer with audio samples.
enum STATE {
READ_INDEX = 0, // Index of the current read position (mod capacity)
WRITE_INDEX, // Index of the current write position (mod capacity)
LENGTH // Clever way of saving the total number of enums values.
}
export class Ring {
state: Int32Array;
channels: Float32Array[];
capacity: number;
constructor(init: RingInit) {
this.state = new Int32Array(init.state)
this.channels = []
for (let channel of init.channels) {
this.channels.push(new Float32Array(channel))
}
this.capacity = init.capacity
}
// Add the samples for single audio frame
write(frame: AudioData): boolean {
let count = frame.numberOfFrames;
let readIndex = Atomics.load(this.state, STATE.READ_INDEX)
let writeIndex = Atomics.load(this.state, STATE.WRITE_INDEX)
let writeIndexNew = writeIndex + count;
// There's not enough space in the ring buffer
if (writeIndexNew - readIndex > this.capacity) {
return false
}
let startIndex = writeIndex % this.capacity;
let endIndex = writeIndexNew % this.capacity;
// Loop over each channel
for (let i = 0; i < this.channels.length; i += 1) {
const channel = this.channels[i]
if (startIndex < endIndex) {
// One continuous range to copy.
const full = channel.subarray(startIndex, endIndex)
frame.copyTo(full, {
planeIndex: i,
frameCount: count,
})
//audio seems to be breaking whenever endIndex is 0
//this works, without "chopiness"
} else if (startIndex >= endIndex && endIndex != 0) {
const first = channel.subarray(startIndex)
const second = channel.subarray(0, endIndex)
frame.copyTo(first, {
planeIndex: i,
frameCount: first.length,
})
//console.log("frame offset", first.length , "frame count", second.length) to test
frame.copyTo(second, {
planeIndex: i,
frameOffset: first.length,
frameCount: second.length,
})
}
}
Atomics.store(this.state, STATE.WRITE_INDEX, writeIndexNew)
return true
}
read(dst: Float32Array[]) {
let readIndex = Atomics.load(this.state, STATE.READ_INDEX)
let writeIndex = Atomics.load(this.state, STATE.WRITE_INDEX)
if (readIndex >= writeIndex) {
// nothing to read
return
}
let readIndexNew = readIndex + dst[0].length
if (readIndexNew > writeIndex) {
// Partial read
readIndexNew = writeIndex
}
let startIndex = readIndex % this.capacity;
let endIndex = readIndexNew % this.capacity;
// Loop over each channel
for (let i = 0; i < dst.length; i += 1) {
if (i >= this.channels.length) {
// ignore excess channels
}
const input = this.channels[i]
const output = dst[i]
if (startIndex < endIndex) {
const full = input.subarray(startIndex, endIndex)
output.set(full)
} else {
const first = input.subarray(startIndex)
const second = input.subarray(0, endIndex)
output.set(first)
output.set(second, first.length)
}
}
Atomics.store(this.state, STATE.READ_INDEX, readIndexNew)
}
// TODO not thread safe
clear() {
const writeIndex = Atomics.load(this.state, STATE.WRITE_INDEX)
Atomics.store(this.state, STATE.READ_INDEX, writeIndex)
}
}
// No prototype to make this easier to send via postMessage
export class RingInit {
state: SharedArrayBuffer;
channels: SharedArrayBuffer[];
capacity: number;
constructor(channels: number, capacity: number) {
// Store the current state in a separate ring buffer.
this.state = new SharedArrayBuffer(STATE.LENGTH * Int32Array.BYTES_PER_ELEMENT)
// Create a buffer for each audio channel
this.channels = []
for (let i = 0; i < channels; i += 1) {
const buffer = new SharedArrayBuffer(capacity * Float32Array.BYTES_PER_ELEMENT)
this.channels.push(buffer)
}
this.capacity = capacity
}
}

26
player/src/audio/worker.ts
View File

@ -1,26 +0,0 @@
import Decoder from "./decoder"
import Renderer from "./renderer"
import * as Message from "./message"
let decoder: Decoder
let renderer: Renderer;
self.addEventListener('message', (e: MessageEvent) => {
if (e.data.config) {
renderer = new Renderer(e.data.config)
decoder = new Decoder(e.data.config, renderer)
}
if (e.data.init) {
decoder.init(e.data.init)
}
if (e.data.segment) {
decoder.decode(e.data.segment)
}
if (e.data.play) {
renderer.play(e.data.play)
}
})

45
player/src/player/index.ts
View File

@ -1,45 +0,0 @@
import Audio from "../audio"
import Transport from "../transport"
import Video from "../video"
export interface PlayerInit {
url: string;
fingerprint?: WebTransportHash; // the certificate fingerprint, temporarily needed for local development
canvas: HTMLCanvasElement;
}
export default class Player {
audio: Audio;
video: Video;
transport: Transport;
constructor(props: PlayerInit) {
this.audio = new Audio()
this.video = new Video({
canvas: props.canvas.transferControlToOffscreen(),
})
this.transport = new Transport({
url: props.url,
fingerprint: props.fingerprint,
audio: this.audio,
video: this.video,
})
}
async close() {
this.transport.close()
}
play() {
this.audio.play({})
//this.video.play()
}
onMessage(msg: any) {
if (msg.sync) {
msg.sync
}
}
}

168
player/src/transport/index.ts
View File

@ -1,168 +0,0 @@
import * as Message from "./message"
import * as Stream from "../stream"
import * as MP4 from "../mp4"
import Audio from "../audio"
import Video from "../video"
export interface TransportInit {
url: string;
fingerprint?: WebTransportHash; // the certificate fingerprint, temporarily needed for local development
audio: Audio;
video: Video;
}
export default class Transport {
quic: Promise<WebTransport>;
api: Promise<WritableStream>;
tracks: Map<string, MP4.InitParser>
audio: Audio;
video: Video;
constructor(props: TransportInit) {
this.tracks = new Map();
this.audio = props.audio;
this.video = props.video;
this.quic = this.connect(props)
// Create a unidirectional stream for all of our messages
this.api = this.quic.then((q) => {
return q.createUnidirectionalStream()
})
// async functions
this.receiveStreams()
}
async close() {
(await this.quic).close()
}
// Helper function to make creating a promise easier
private async connect(props: TransportInit): Promise<WebTransport> {
let options: WebTransportOptions = {};
if (props.fingerprint) {
options.serverCertificateHashes = [ props.fingerprint ]
}
const quic = new WebTransport(props.url, options)
await quic.ready
return quic
}
async sendMessage(msg: any) {
const payload = JSON.stringify(msg)
const size = payload.length + 8
const stream = await this.api
const writer = new Stream.Writer(stream)
await writer.uint32(size)
await writer.string("warp")
await writer.string(payload)
writer.release()
}
async receiveStreams() {
const q = await this.quic
const streams = q.incomingUnidirectionalStreams.getReader()
while (true) {
const result = await streams.read()
if (result.done) break
const stream = result.value
this.handleStream(stream) // don't await
}
}
async handleStream(stream: ReadableStream) {
let r = new Stream.Reader(stream)
while (!await r.done()) {
const size = await r.uint32();
const typ = new TextDecoder('utf-8').decode(await r.bytes(4));
if (typ != "warp") throw "expected warp atom"
if (size < 8) throw "atom too small"
const payload = new TextDecoder('utf-8').decode(await r.bytes(size - 8));
const msg = JSON.parse(payload)
if (msg.init) {
return this.handleInit(r, msg.init as Message.Init)
} else if (msg.segment) {
return this.handleSegment(r, msg.segment as Message.Segment)
}
}
}
async handleInit(stream: Stream.Reader, msg: Message.Init) {
let track = this.tracks.get(msg.id);
if (!track) {
track = new MP4.InitParser()
this.tracks.set(msg.id, track)
}
while (1) {
const data = await stream.read()
if (!data) break
track.push(data)
}
const info = await track.info
if (info.audioTracks.length + info.videoTracks.length != 1) {
throw new Error("expected a single track")
}
if (info.audioTracks.length) {
this.audio.init({
track: msg.id,
info: info,
raw: track.raw,
})
} else if (info.videoTracks.length) {
this.video.init({
track: msg.id,
info: info,
raw: track.raw,
})
} else {
throw new Error("init is neither audio nor video")
}
}
async handleSegment(stream: Stream.Reader, msg: Message.Segment) {
let track = this.tracks.get(msg.init);
if (!track) {
track = new MP4.InitParser()
this.tracks.set(msg.init, track)
}
// Wait until we learn if this is an audio or video track
const info = await track.info
if (info.audioTracks.length) {
this.audio.segment({
track: msg.init,
buffer: stream.buffer,
reader: stream.reader,
})
} else if (info.videoTracks.length) {
this.video.segment({
track: msg.init,
buffer: stream.buffer,
reader: stream.reader,
})
} else {
throw new Error("segment is neither audio nor video")
}
}
}

13
player/src/transport/message.ts
View File

@ -1,13 +0,0 @@
export interface Init {
id: string
}
export interface Segment {
init: string // id of the init segment
timestamp: number // presentation timestamp in milliseconds of the first sample
// TODO track would be nice
}
export interface Debug {
max_bitrate: number
}

127
player/src/video/decoder.ts
View File

@ -1,127 +0,0 @@
import * as Message from "./message";
import * as MP4 from "../mp4"
import * as Stream from "../stream"
import * as Util from "../util"
import Renderer from "./renderer"
export default class Decoder {
// Store the init message for each track
tracks: Map<string, Util.Deferred<Message.Init>>
renderer: Renderer;
constructor(renderer: Renderer) {
this.tracks = new Map();
this.renderer = renderer;
}
async init(msg: Message.Init) {
let track = this.tracks.get(msg.track);
if (!track) {
track = new Util.Deferred()
this.tracks.set(msg.track, track)
}
if (msg.info.videoTracks.length != 1 || msg.info.audioTracks.length != 0) {
throw new Error("Expected a single video track")
}
track.resolve(msg)
}
async decode(msg: Message.Segment) {
let track = this.tracks.get(msg.track);
if (!track) {
track = new Util.Deferred()
this.tracks.set(msg.track, track)
}
// Wait for the init segment to be fully received and parsed
const init = await track.promise;
const info = init.info;
const video = info.videoTracks[0]
const decoder = new VideoDecoder({
output: (frame: VideoFrame) => {
this.renderer.emit(frame)
},
error: (err: Error) => {
console.warn(err)
}
});
const input = MP4.New();
input.onSamples = (id: number, user: any, samples: MP4.Sample[]) => {
for (let sample of samples) {
const timestamp = 1000 * sample.dts / sample.timescale // milliseconds
if (sample.is_sync) {
// Configure the decoder using the AVC box for H.264
const avcc = sample.description.avcC;
const description = new MP4.Stream(new Uint8Array(avcc.size), 0, false)
avcc.write(description)
decoder.configure({
codec: video.codec,
codedHeight: video.track_height,
codedWidth: video.track_width,
description: description.buffer?.slice(8),
// optimizeForLatency: true
})
}
decoder.decode(new EncodedVideoChunk({
data: sample.data,
duration: sample.duration,
timestamp: timestamp,
type: sample.is_sync ? "key" : "delta",
}))
}
}
input.onReady = (info: any) => {
input.setExtractionOptions(info.tracks[0].id, {}, { nbSamples: 1 });
input.start();
}
// MP4box requires us to reparse the init segment unfortunately
let offset = 0;
for (let raw of init.raw) {
raw.fileStart = offset
input.appendBuffer(raw)
}
const stream = new Stream.Reader(msg.reader, msg.buffer)
/* TODO I'm not actually sure why this code doesn't work; something trips up the MP4 parser
while (1) {
const data = await stream.read()
if (!data) break
input.appendBuffer(data)
input.flush()
}
*/
// One day I'll figure it out; until then read one top-level atom at a time
while (!await stream.done()) {
const raw = await stream.peek(4)
const size = new DataView(raw.buffer, raw.byteOffset, raw.byteLength).getUint32(0)
const atom = await stream.bytes(size)
// Make a copy of the atom because mp4box only accepts an ArrayBuffer unfortunately
let box = new Uint8Array(atom.byteLength);
box.set(atom)
// and for some reason we need to modify the underlying ArrayBuffer with offset
let buffer = box.buffer as MP4.ArrayBuffer
buffer.fileStart = offset
// Parse the data
offset = input.appendBuffer(buffer)
input.flush()
}
}
}

27
player/src/video/index.ts
View File

@ -1,27 +0,0 @@
import * as Message from "./message"
// Wrapper around the WebWorker API
export default class Video {
worker: Worker;
constructor(config: Message.Config) {
const url = new URL('worker.ts', import.meta.url)
this.worker = new Worker(url, {
type: "module",
name: "video",
})
this.worker.postMessage({ config }, [ config.canvas ])
}
init(init: Message.Init) {
this.worker.postMessage({ init }) // note: we copy the raw init bytes each time
}
segment(segment: Message.Segment) {
this.worker.postMessage({ segment }, [ segment.buffer.buffer, segment.reader ])
}
play() {
// TODO
}
}

17
player/src/video/message.ts
View File

@ -1,17 +0,0 @@
import * as MP4 from "../mp4"
export interface Config {
canvas: OffscreenCanvas;
}
export interface Init {
track: string;
info: MP4.Info;
raw: MP4.ArrayBuffer[];
}
export interface Segment {
track: string;
buffer: Uint8Array; // unread buffered data
reader: ReadableStream; // unread unbuffered data
}

1
server/.dockerignore Normal file
View File

@ -0,0 +1 @@
target

2
server/.gitignore vendored
View File

@ -1 +1 @@
logs/
target

3
server/.vscode/settings.json vendored Normal file
View File

@ -0,0 +1,3 @@
{
"rust-analyzer.showUnlinkedFileNotification": false
}

1055
server/Cargo.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

18
server/Cargo.toml Normal file
View File

@ -0,0 +1,18 @@
[package]
name = "warp"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
quiche = { git = "https://github.com/kixelated/quiche.git", branch = "master", features = [ "qlog" ] } # WebTransport fork
clap = { version = "4.0", features = [ "derive" ] }
log = { version = "0.4", features = ["std"] }
mio = { version = "0.8", features = ["net", "os-poll"] }
env_logger = "0.9.3"
ring = "0.16"
anyhow = "1.0.70"
mp4 = "0.13.0"
serde = "1.0.160"
serde_json = "1.0"

42
server/Dockerfile Normal file
View File

@ -0,0 +1,42 @@
# Use the official Rust image as the base image
FROM rust:latest as build
# Quiche requires docker
RUN apt-get update && \
apt-get install -y cmake
# Set the build directory
WORKDIR /warp
# Create an empty project
RUN cargo init --bin
# Copy the Cargo.toml and Cargo.lock files to the container
COPY Cargo.toml Cargo.lock ./
# Build the empty project so we download/cache dependencies
RUN cargo build --release
# Copy the entire project to the container
COPY . .
# Build the project
RUN cargo build --release
# Make a new image to run the binary
FROM ubuntu:latest
# Use a volume to access certificates
VOLUME /cert
# Use another volume to access the media
VOLUME /media
# Expose port 4443 for the server
EXPOSE 4443/udp
# Copy the built binary
COPY --from=build /warp/target/release/warp /bin
# Set the startup command to run the binary
CMD warp --cert /cert/localhost.crt --key /cert/localhost.key --media /media/fragmented.mp4

30
server/go.mod
View File

@ -1,30 +0,0 @@
module github.com/kixelated/warp/server
go 1.18
require (
github.com/abema/go-mp4 v0.7.2
github.com/kixelated/invoker v1.0.0
github.com/kixelated/quic-go v1.31.0
github.com/kixelated/webtransport-go v1.4.1
github.com/zencoder/go-dash/v3 v3.0.2
)
require (
github.com/francoispqt/gojay v1.2.13 // indirect
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 // indirect
github.com/google/uuid v1.1.2 // indirect
github.com/marten-seemann/qpack v0.3.0 // indirect
github.com/marten-seemann/qtls-go1-18 v0.1.3 // indirect
github.com/marten-seemann/qtls-go1-19 v0.1.1 // indirect
github.com/onsi/ginkgo/v2 v2.2.0 // indirect
golang.org/x/crypto v0.0.0-20220331220935-ae2d96664a29 // indirect
golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e // indirect
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 // indirect
golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
golang.org/x/sys v0.1.1-0.20221102194838-fc697a31fa06 // indirect
golang.org/x/text v0.3.7 // indirect
golang.org/x/tools v0.1.12 // indirect
)

264
server/go.sum
View File

@ -1,264 +0,0 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.31.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.37.0/go.mod h1:TS1dMSSfndXH133OKGwekG838Om/cQT0BUHV3HcBgoo=
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3/go.mod h1:Yl+fi1br7+Rr3LqpNJf1/uxUdtRUV+Tnj0o93V2B9MU=
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0/go.mod h1:JLBrvjyP0v+ecvNYvCpyZgu5/xkfAUhi6wJj28eUfSU=
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412/go.mod h1:a1inKt/atXimZ4Mv927x+r7UpyzRUf4emIoiiSC2TN4=
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c/go.mod h1:0PRwlb0D6DFvNNtx+9ybjezNCa8XF0xaYcETyp6rHWU=
git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/abema/go-mp4 v0.7.2 h1:ugTC8gfEmjyaDKpXs3vi2QzgJbDu9B8m6UMMIpbYbGg=
github.com/abema/go-mp4 v0.7.2/go.mod h1:vPl9t5ZK7K0x68jh12/+ECWBCXoWuIDtNgPtU2f04ws=
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g=
github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
github.com/francoispqt/gojay v1.2.13 h1:d2m3sFjloqoIUQU3TsHBgj6qg/BVGlTBeHDUmyJnXKk=
github.com/francoispqt/gojay v1.2.13/go.mod h1:ehT5mTG4ua4581f1++1WLG0vPdaA9HaiDsoyrBGkyDY=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 h1:p104kn46Q8WdvHunIJ9dAyjPVtrBPhSr3KT2yUst43I=
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.6.0 h1:ErTB+efbowRARo13NNdxyJji2egdxLGQhRaY+DUumQc=
github.com/golang/mock v1.6.0/go.mod h1:p6yTPP+5HYm5mzsMV8JkE6ZKdX+/wYM6Hr+LicevLPs=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
github.com/google/go-github v17.0.0+incompatible/go.mod h1:zLgOLi98H3fifZn+44m+umXrS52loVEgC2AApnigrVQ=
github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 h1:yAJXTCF9TqKcTiHJAE8dj7HMvPfh66eeA2JYW7eFpSE=
github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go v2.0.0+incompatible/go.mod h1:SFVmujtThgffbyetf+mdk2eWhX2bMyUtNHzFKcPA9HY=
github.com/googleapis/gax-go/v2 v2.0.3/go.mod h1:LLvjysVCY1JZeum8Z6l8qUty8fiNwE08qbEPm1M08qg=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
github.com/grpc-ecosystem/grpc-gateway v1.5.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/kisielk/errcheck v1.4.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kixelated/invoker v1.0.0 h1:0wYlvK39yQPbkwIFy+YN41AhF89WOtGyWqV2pZB39xw=
github.com/kixelated/invoker v1.0.0/go.mod h1:RjG3iqm/sKwZjOpcW4SGq+l+4DJCDR/yUtc70VjCRB8=
github.com/kixelated/quic-go v1.31.0 h1:p2vq3Otvtmz+0EP23vjumnO/HU4Q/DFxNF6xNryVfmA=
github.com/kixelated/quic-go v1.31.0/go.mod h1:AO7pURnb8HXHmdalp5e09UxQfsuwseEhl0NLmwiSOFY=
github.com/kixelated/webtransport-go v1.4.1 h1:ZtY3P7hVe1wK5fAt71b+HHnNISFDcQ913v+bvaNATxA=
github.com/kixelated/webtransport-go v1.4.1/go.mod h1:6RV5pTXF7oP53T83bosSDsLdSdw31j5cfpMDqsO4D5k=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.3/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI=
github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/marten-seemann/qpack v0.3.0 h1:UiWstOgT8+znlkDPOg2+3rIuYXJ2CnGDkGUXN6ki6hE=
github.com/marten-seemann/qpack v0.3.0/go.mod h1:cGfKPBiP4a9EQdxCwEwI/GEeWAsjSekBvx/X8mh58+g=
github.com/marten-seemann/qtls-go1-18 v0.1.3 h1:R4H2Ks8P6pAtUagjFty2p7BVHn3XiwDAl7TTQf5h7TI=
github.com/marten-seemann/qtls-go1-18 v0.1.3/go.mod h1:mJttiymBAByA49mhlNZZGrH5u1uXYZJ+RW28Py7f4m4=
github.com/marten-seemann/qtls-go1-19 v0.1.1 h1:mnbxeq3oEyQxQXwI4ReCgW9DPoPR94sNlqWoDZnjRIE=
github.com/marten-seemann/qtls-go1-19 v0.1.1/go.mod h1:5HTDWtVudo/WFsHKRNuOhWlbdjrfs5JHrYb0wIJqGpI=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/microcosm-cc/bluemonday v1.0.1/go.mod h1:hsXNsILzKxV+sX77C5b8FSuKF00vh2OMYv+xgHpAMF4=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo=
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM=
github.com/onsi/ginkgo/v2 v2.2.0 h1:3ZNA3L1c5FYDFTTxbFeVGGD8jYvjYauHD30YgLxVsNI=
github.com/onsi/ginkgo/v2 v2.2.0/go.mod h1:MEH45j8TBi6u9BMogfbp0stKC5cdGjumZj5Y7AG4VIk=
github.com/onsi/gomega v1.20.1 h1:PA/3qinGoukvymdIDV8pii6tiZgC8kbmJO6Z5+b002Q=
github.com/openzipkin/zipkin-go v0.1.1/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTmOf0Erfk+hxe8=
github.com/orcaman/writerseeker v0.0.0-20200621085525-1d3f536ff85e h1:s2RNOM/IGdY0Y6qfTeUKhDawdHDpK9RGBdx80qN4Ttw=
github.com/orcaman/writerseeker v0.0.0-20200621085525-1d3f536ff85e/go.mod h1:nBdnFKj15wFbf94Rwfq4m30eAcyY9V/IyKAGQFtqkW0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v0.8.0/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4/go.mod h1:XhFIlyj5a1fBNx5aJTbKoIq0mNaPvOagO+HjB3EtxrY=
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48/go.mod h1:5u70Mqkb5O5cxEA8nxTsgrgLehJeAw6Oc4Ab1c/P1HM=
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470/go.mod h1:2dOwnU2uBioM+SGy2aZoq1f/Sd1l9OkAeAUvjSyvgU0=
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk=
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041/go.mod h1:N5mDOmsrJOB+vfqUK+7DmDyjhSLIIBnXo9lvZJj3MWQ=
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d/go.mod h1:05UtEgK5zq39gLST6uB0cf3NEHjETfB4Fgr3Gx5R9Vw=
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c/go.mod h1:8d3azKNyqcHP1GaQE/c6dDgjkgSx2BZ4IoEi4F1reUI=
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b/go.mod h1:ZpfEhSmds4ytuByIcDnOLkTHGUI6KNqRNPDLHDk+mUU=
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20/go.mod h1:UDKB5a1T23gOMUJrI+uSuH0VRDStOiUVSjBTRDVBVag=
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9/go.mod h1:+rgNQw2P9ARFAs37qieuu7ohDNQ3gds9msbT2yn85sg=
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50/go.mod h1:zPn1wHpTIePGnXSHpsVPWEktKXHr6+SS6x/IKRb7cpw=
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc/go.mod h1:aYMfkZ6DWSJPJ6c4Wwz3QtW22G7mf/PEgaB9k/ik5+Y=
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371/go.mod h1:ZY1cvUeJuFPAdZ/B6v7RHavJWZn2YPVFQ1OSXhCGOkg=
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9/go.mod h1:919LwcH0M7/W4fcZ0/jy0qGght1GIhqyS/EgWGH2j5Q=
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191/go.mod h1:e2qWDig5bLteJ4fwvDAc2NHzqFEthkqn7aOZAOpj+PQ=
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241/go.mod h1:NPpHK2TI7iSaM0buivtFUc9offApnI0Alt/K8hcHy0I=
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122/go.mod h1:b5uSkrEVM1jQUspwbixRBhaIjIzL2xazXp6kntxYle0=
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2/go.mod h1:eWdoE5JD4R5UVWDucdOPg1g2fqQRq78IQa9zlOV1vpQ=
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82/go.mod h1:TCR1lToEk4d2s07G3XGfz2QrgHXg4RJBvjrOozvoWfk=
github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537/go.mod h1:QJTqeLYEDaXHZDBsXlPCDqdhQuJkuw4NOtaxYe3xii4=
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133/go.mod h1:hKmq5kWdCj2z2KEozexVbfEZIWiTjhE0+UjmZgPqehw=
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d/go.mod h1:UdhH50NIW0fCiwBSr0co2m7BnFLdv4fQTgdqdJTHFeE=
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e/go.mod h1:HuIsMU8RRBOtsCgI77wP899iHVBQpCmg4ErYMZB+2IA=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/sunfish-shogi/bufseekio v0.0.0-20210207115823-a4185644b365/go.mod h1:dEzdXgvImkQ3WLI+0KQpmEx8T/C/ma9KeS3AfmU899I=
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/zencoder/go-dash/v3 v3.0.2 h1:oP1+dOh+Gp57PkvdCyMfbHtrHaxfl3w4kR3KBBbuqQE=
github.com/zencoder/go-dash/v3 v3.0.2/go.mod h1:30R5bKy1aUYY45yesjtZ9l8trNc2TwNqbS17WVQmCzk=
go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA=
go4.org v0.0.0-20180809161055-417644f6feb5/go.mod h1:MkTOUMDaeVYJUOUsaDXIhWPZYa1yOyC1qaOBpL57BhE=
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d/go.mod h1:OWs+y06UdEOHN4y+MfF/py+xQ/tYqIWW03b70/CG9Rw=
golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20220331220935-ae2d96664a29 h1:tkVvjkPTB7pnW3jnid7kNyAMPVWllTNOf/qKDze4p9o=
golang.org/x/crypto v0.0.0-20220331220935-ae2d96664a29/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e h1:+WEEuIdZHnUeJJmEUjyYC2gfUMj69yZXw17EnHg/otA=
golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e/go.mod h1:Kr81I6Kryrl9sr8s2FK3vxD90NdsKWRuOIl2O4CvYbA=
golang.org/x/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181029044818-c44066c5c816/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181106065722-10aee1819953/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190313220215-9f648a60d977/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852/go.mod h1:JLpeXjPJfIyPr5TlbXLkXWLhP8nz10XfvxElABhCtcw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181029174526-d69651ed3497/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190316082340-a2f829d7f35f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.1-0.20221102194838-fc697a31fa06 h1:E1pm64FqQa4v8dHd/bAneyMkR4hk8LTJhoSlc5mc1cM=
golang.org/x/sys v0.1.1-0.20221102194838-fc697a31fa06/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030000716-a0a13e073c7b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200410194907-79a7a3126eef/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.12 h1:VveCTK38A2rkS8ZqFY25HIDFscX5X9OoEhJd3quQmXU=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.0.0-20181030000543-1d582fd0359e/go.mod h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
google.golang.org/api v0.1.0/go.mod h1:UGEZY7KEX120AnNLIHFMKIo4obdJhkp2tPbaPlQx13Y=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20180831171423-11092d34479b/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20181202183823-bd91e49a0898/go.mod h1:7Ep/1NZk928CDR8SjdVbjWNpdIf6nzjE3BTgJDr2Atg=
google.golang.org/genproto v0.0.0-20190306203927-b5d61aea6440/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.16.0/go.mod h1:0JHn/cJsOMiMfNA9+DeHDlAU7KAAB5GDlYFpa9MZMio=
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/src-d/go-billy.v4 v4.3.2 h1:0SQA1pRztfTFx2miS8sA97XvooFeNOmvUenF4o0EcVg=
gopkg.in/src-d/go-billy.v4 v4.3.2/go.mod h1:nDjArDMp+XMs1aFAESLRjfGSgfvoYN0hDfzEk0GjC98=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919/go.mod h1:77eQGdRu53HpSqPFJFmuJdjuHRquDANNeA4x7B8WQ9o=
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2020.1.6/go.mod h1:pyyisuGw24ruLjrr1ddx39WE0y9OooInRzEYLhQB2YY=
sourcegraph.com/sourcegraph/go-diff v0.5.0/go.mod h1:kuch7UrkMzY0X+p9CRK03kfuPQ2zzQcaEFbx8wA8rck=
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4/go.mod h1:ketZ/q3QxT9HOBeFhu6RdvsftgpsbFHBF5Cas6cDKZ0=

380
server/internal/warp/media.go
View File

@ -1,380 +0,0 @@
package warp
import (
"bytes"
"context"
"encoding/binary"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"strings"
"time"
"github.com/abema/go-mp4"
"github.com/kixelated/invoker"
"github.com/zencoder/go-dash/v3/mpd"
)
// This is a demo; you should actually fetch media from a live backend.
// It's just much easier to read from disk and "fake" being live.
type Media struct {
base fs.FS
inits map[string]*MediaInit
video []*mpd.Representation
audio []*mpd.Representation
}
func NewMedia(playlistPath string) (m *Media, err error) {
m = new(Media)
// Create a fs.FS out of the folder holding the playlist
m.base = os.DirFS(filepath.Dir(playlistPath))
// Read the playlist file
playlist, err := mpd.ReadFromFile(playlistPath)
if err != nil {
return nil, fmt.Errorf("failed to open playlist: %w", err)
}
if len(playlist.Periods) > 1 {
return nil, fmt.Errorf("multiple periods not supported")
}
period := playlist.Periods[0]
for _, adaption := range period.AdaptationSets {
representation := adaption.Representations[0]
if representation.MimeType == nil {
return nil, fmt.Errorf("missing representation mime type")
}
if representation.Bandwidth == nil {
return nil, fmt.Errorf("missing representation bandwidth")
}
switch *representation.MimeType {
case "video/mp4":
m.video = append(m.video, representation)
case "audio/mp4":
m.audio = append(m.audio, representation)
}
}
if len(m.video) == 0 {
return nil, fmt.Errorf("no video representation found")
}
if len(m.audio) == 0 {
return nil, fmt.Errorf("no audio representation found")
}
m.inits = make(map[string]*MediaInit)
var reps []*mpd.Representation
reps = append(reps, m.audio...)
reps = append(reps, m.video...)
for _, rep := range reps {
path := *rep.SegmentTemplate.Initialization
// TODO Support the full template engine
path = strings.ReplaceAll(path, "$RepresentationID$", *rep.ID)
f, err := fs.ReadFile(m.base, path)
if err != nil {
return nil, fmt.Errorf("failed to read init file: %w", err)
}
init, err := newMediaInit(*rep.ID, f)
if err != nil {
return nil, fmt.Errorf("failed to create init segment: %w", err)
}
m.inits[*rep.ID] = init
}
return m, nil
}
func (m *Media) Start(bitrate func() uint64) (inits map[string]*MediaInit, audio *MediaStream, video *MediaStream, err error) {
start := time.Now()
audio, err = newMediaStream(m, m.audio, start, bitrate)
if err != nil {
return nil, nil, nil, err
}
video, err = newMediaStream(m, m.video, start, bitrate)
if err != nil {
return nil, nil, nil, err
}
return m.inits, audio, video, nil
}
type MediaStream struct {
Media *Media
start time.Time
reps []*mpd.Representation
sequence int
bitrate func() uint64 // returns the current estimated bitrate
}
func newMediaStream(m *Media, reps []*mpd.Representation, start time.Time, bitrate func() uint64) (ms *MediaStream, err error) {
ms = new(MediaStream)
ms.Media = m
ms.reps = reps
ms.start = start
ms.bitrate = bitrate
return ms, nil
}
func (ms *MediaStream) chooseRepresentation() (choice *mpd.Representation) {
bitrate := ms.bitrate()
// Loop over the renditions and pick the highest bitrate we can support
for _, r := range ms.reps {
if uint64(*r.Bandwidth) <= bitrate && (choice == nil || *r.Bandwidth > *choice.Bandwidth) {
choice = r
}
}
if choice != nil {
return choice
}
// We can't support any of the bitrates, so find the lowest one.
for _, r := range ms.reps {
if choice == nil || *r.Bandwidth < *choice.Bandwidth {
choice = r
}
}
return choice
}
// Returns the next segment in the stream
func (ms *MediaStream) Next(ctx context.Context) (segment *MediaSegment, err error) {
rep := ms.chooseRepresentation()
if rep.SegmentTemplate == nil {
return nil, fmt.Errorf("missing segment template")
}
if rep.SegmentTemplate.Media == nil {
return nil, fmt.Errorf("no media template")
}
if rep.SegmentTemplate.StartNumber == nil {
return nil, fmt.Errorf("missing start number")
}
path := *rep.SegmentTemplate.Media
sequence := ms.sequence + int(*rep.SegmentTemplate.StartNumber)
// TODO Support the full template engine
path = strings.ReplaceAll(path, "$RepresentationID$", *rep.ID)
path = strings.ReplaceAll(path, "$Number%05d$", fmt.Sprintf("%05d", sequence)) // TODO TODO
// Try openning the file
f, err := ms.Media.base.Open(path)
if errors.Is(err, os.ErrNotExist) && ms.sequence != 0 {
// Return EOF if the next file is missing
return nil, nil
} else if err != nil {
return nil, fmt.Errorf("failed to open segment file: %w", err)
}
duration := time.Duration(*rep.SegmentTemplate.Duration) / time.Nanosecond
timestamp := time.Duration(ms.sequence) * duration
init := ms.Media.inits[*rep.ID]
segment, err = newMediaSegment(ms, init, f, timestamp)
if err != nil {
return nil, fmt.Errorf("failed to create segment: %w", err)
}
ms.sequence += 1
return segment, nil
}
type MediaInit struct {
ID string
Raw []byte
Timescale int
}
func newMediaInit(id string, raw []byte) (mi *MediaInit, err error) {
mi = new(MediaInit)
mi.ID = id
mi.Raw = raw
err = mi.parse()
if err != nil {
return nil, fmt.Errorf("failed to parse init segment: %w", err)
}
return mi, nil
}
// Parse through the init segment, literally just to populate the timescale
func (mi *MediaInit) parse() (err error) {
r := bytes.NewReader(mi.Raw)
_, err = mp4.ReadBoxStructure(r, func(h *mp4.ReadHandle) (interface{}, error) {
if !h.BoxInfo.IsSupportedType() {
return nil, nil
}
payload, _, err := h.ReadPayload()
if err != nil {
return nil, err
}
switch box := payload.(type) {
case *mp4.Mdhd: // Media Header; moov -> trak -> mdia > mdhd
if mi.Timescale != 0 {
// verify only one track
return nil, fmt.Errorf("multiple mdhd atoms")
}
mi.Timescale = int(box.Timescale)
}
// Expands children
return h.Expand()
})
if err != nil {
return fmt.Errorf("failed to parse MP4 file: %w", err)
}
return nil
}
type MediaSegment struct {
Stream *MediaStream
Init *MediaInit
file fs.File
timestamp time.Duration
}
func newMediaSegment(s *MediaStream, init *MediaInit, file fs.File, timestamp time.Duration) (ms *MediaSegment, err error) {
ms = new(MediaSegment)
ms.Stream = s
ms.Init = init
ms.file = file
ms.timestamp = timestamp
return ms, nil
}
// Return the next atom, sleeping based on the PTS to simulate a live stream
func (ms *MediaSegment) Read(ctx context.Context) (chunk []byte, err error) {
// Read the next top-level box
var header [8]byte
_, err = io.ReadFull(ms.file, header[:])
if err != nil {
return nil, fmt.Errorf("failed to read header: %w", err)
}
size := int(binary.BigEndian.Uint32(header[0:4]))
if size < 8 {
return nil, fmt.Errorf("box is too small")
}
buf := make([]byte, size)
n := copy(buf, header[:])
_, err = io.ReadFull(ms.file, buf[n:])
if err != nil {
return nil, fmt.Errorf("failed to read atom: %w", err)
}
sample, err := ms.parseAtom(ctx, buf)
if err != nil {
return nil, fmt.Errorf("failed to parse atom: %w", err)
}
if sample != nil {
// Simulate a live stream by sleeping before we write this sample.
// Figure out how much time has elapsed since the start
elapsed := time.Since(ms.Stream.start)
delay := sample.Timestamp - elapsed
if delay > 0 {
// Sleep until we're supposed to see these samples
err = invoker.Sleep(delay)(ctx)
if err != nil {
return nil, err
}
}
}
return buf, nil
}
// Parse through the MP4 atom, returning infomation about the next fragmented sample
func (ms *MediaSegment) parseAtom(ctx context.Context, buf []byte) (sample *mediaSample, err error) {
r := bytes.NewReader(buf)
_, err = mp4.ReadBoxStructure(r, func(h *mp4.ReadHandle) (interface{}, error) {
if !h.BoxInfo.IsSupportedType() {
return nil, nil
}
payload, _, err := h.ReadPayload()
if err != nil {
return nil, err
}
switch box := payload.(type) {
case *mp4.Moof:
sample = new(mediaSample)
case *mp4.Tfdt: // Track Fragment Decode Timestamp; moof -> traf -> tfdt
// TODO This box isn't required
// TODO we want the last PTS if there are multiple samples
var dts time.Duration
if box.FullBox.Version == 0 {
dts = time.Duration(box.BaseMediaDecodeTimeV0)
} else {
dts = time.Duration(box.BaseMediaDecodeTimeV1)
}
if ms.Init.Timescale == 0 {
return nil, fmt.Errorf("missing timescale")
}
// Convert to seconds
// TODO What about PTS?
sample.Timestamp = dts * time.Second / time.Duration(ms.Init.Timescale)
}
// Expands children
return h.Expand()
})
if err != nil {
return nil, fmt.Errorf("failed to parse MP4 file: %w", err)
}
return sample, nil
}
func (ms *MediaSegment) Close() (err error) {
return ms.file.Close()
}
type mediaSample struct {
Timestamp time.Duration // The timestamp of the first sample
}

20
server/internal/warp/message.go
View File

@ -1,20 +0,0 @@
package warp
type Message struct {
Init *MessageInit `json:"init,omitempty"`
Segment *MessageSegment `json:"segment,omitempty"`
Debug *MessageDebug `json:"debug,omitempty"`
}
type MessageInit struct {
Id string `json:"id"` // ID of the init segment
}
type MessageSegment struct {
Init string `json:"init"` // ID of the init segment to use for this segment
Timestamp int `json:"timestamp"` // PTS of the first frame in milliseconds
}
type MessageDebug struct {
MaxBitrate int `json:"max_bitrate"` // Artificially limit the QUIC max bitrate
}

132
server/internal/warp/server.go
View File

@ -1,132 +0,0 @@
package warp
import (
"context"
"crypto/tls"
"encoding/hex"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
"github.com/kixelated/invoker"
"github.com/kixelated/quic-go"
"github.com/kixelated/quic-go/http3"
"github.com/kixelated/quic-go/logging"
"github.com/kixelated/quic-go/qlog"
"github.com/kixelated/webtransport-go"
)
type Server struct {
inner *webtransport.Server
media *Media
sessions invoker.Tasks
cert *tls.Certificate
}
type Config struct {
Addr string
Cert *tls.Certificate
LogDir string
Media *Media
}
func New(config Config) (s *Server, err error) {
s = new(Server)
s.cert = config.Cert
s.media = config.Media
quicConfig := &quic.Config{}
if config.LogDir != "" {
quicConfig.Tracer = qlog.NewTracer(func(p logging.Perspective, connectionID []byte) io.WriteCloser {
path := fmt.Sprintf("%s-%s.qlog", p, hex.EncodeToString(connectionID))
f, err := os.Create(filepath.Join(config.LogDir, path))
if err != nil {
// lame
panic(err)
}
return f
})
}
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{*s.cert},
}
// Host a HTTP/3 server to serve the WebTransport endpoint
mux := http.NewServeMux()
mux.HandleFunc("/watch", s.handleWatch)
s.inner = &webtransport.Server{
H3: http3.Server{
TLSConfig: tlsConfig,
QuicConfig: quicConfig,
Addr: config.Addr,
Handler: mux,
},
CheckOrigin: func(r *http.Request) bool { return true },
}
return s, nil
}
func (s *Server) runServe(ctx context.Context) (err error) {
return s.inner.ListenAndServe()
}
func (s *Server) runShutdown(ctx context.Context) (err error) {
<-ctx.Done()
s.inner.Close() // close on context shutdown
return ctx.Err()
}
func (s *Server) Run(ctx context.Context) (err error) {
return invoker.Run(ctx, s.runServe, s.runShutdown, s.sessions.Repeat)
}
func (s *Server) handleWatch(w http.ResponseWriter, r *http.Request) {
hijacker, ok := w.(http3.Hijacker)
if !ok {
panic("unable to hijack connection: must use kixelated/quic-go")
}
conn := hijacker.Connection()
sess, err := s.inner.Upgrade(w, r)
if err != nil {
http.Error(w, "failed to upgrade session", 500)
return
}
err = s.serveSession(r.Context(), conn, sess)
if err != nil {
log.Println(err)
}
}
func (s *Server) serveSession(ctx context.Context, conn quic.Connection, sess *webtransport.Session) (err error) {
defer func() {
if err != nil {
sess.CloseWithError(1, err.Error())
} else {
sess.CloseWithError(0, "end of broadcast")
}
}()
ss, err := NewSession(conn, sess, s.media)
if err != nil {
return fmt.Errorf("failed to create session: %w", err)
}
err = ss.Run(ctx)
if err != nil {
return fmt.Errorf("terminated session: %w", err)
}
return nil
}

279
server/internal/warp/session.go
View File

@ -1,279 +0,0 @@
package warp
import (
"context"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"math"
"time"
"github.com/kixelated/invoker"
"github.com/kixelated/quic-go"
"github.com/kixelated/webtransport-go"
)
// A single WebTransport session
type Session struct {
conn quic.Connection
inner *webtransport.Session
media *Media
inits map[string]*MediaInit
audio *MediaStream
video *MediaStream
streams invoker.Tasks
}
func NewSession(connection quic.Connection, session *webtransport.Session, media *Media) (s *Session, err error) {
s = new(Session)
s.conn = connection
s.inner = session
s.media = media
return s, nil
}
func (s *Session) Run(ctx context.Context) (err error) {
s.inits, s.audio, s.video, err = s.media.Start(s.conn.GetMaxBandwidth)
if err != nil {
return fmt.Errorf("failed to start media: %w", err)
}
// Once we've validated the session, now we can start accessing the streams
return invoker.Run(ctx, s.runAccept, s.runAcceptUni, s.runInit, s.runAudio, s.runVideo, s.streams.Repeat)
}
func (s *Session) runAccept(ctx context.Context) (err error) {
for {
stream, err := s.inner.AcceptStream(ctx)
if err != nil {
return fmt.Errorf("failed to accept bidirectional stream: %w", err)
}
// Warp doesn't utilize bidirectional streams so just close them immediately.
// We might use them in the future so don't close the connection with an error.
stream.CancelRead(1)
}
}
func (s *Session) runAcceptUni(ctx context.Context) (err error) {
for {
stream, err := s.inner.AcceptUniStream(ctx)
if err != nil {
return fmt.Errorf("failed to accept unidirectional stream: %w", err)
}
s.streams.Add(func(ctx context.Context) (err error) {
return s.handleStream(ctx, stream)
})
}
}
func (s *Session) handleStream(ctx context.Context, stream webtransport.ReceiveStream) (err error) {
defer func() {
if err != nil {
stream.CancelRead(1)
}
}()
var header [8]byte
for {
_, err = io.ReadFull(stream, header[:])
if errors.Is(io.EOF, err) {
return nil
} else if err != nil {
return fmt.Errorf("failed to read atom header: %w", err)
}
size := binary.BigEndian.Uint32(header[0:4])
name := string(header[4:8])
if size < 8 {
return fmt.Errorf("atom size is too small")
} else if size > 42069 { // arbitrary limit
return fmt.Errorf("atom size is too large")
} else if name != "warp" {
return fmt.Errorf("only warp atoms are supported")
}
payload := make([]byte, size-8)
_, err = io.ReadFull(stream, payload)
if err != nil {
return fmt.Errorf("failed to read atom payload: %w", err)
}
log.Println("received message:", string(payload))
msg := Message{}
err = json.Unmarshal(payload, &msg)
if err != nil {
return fmt.Errorf("failed to decode json payload: %w", err)
}
if msg.Debug != nil {
s.setDebug(msg.Debug)
}
}
}
func (s *Session) runInit(ctx context.Context) (err error) {
for _, init := range s.inits {
err = s.writeInit(ctx, init)
if err != nil {
return fmt.Errorf("failed to write init stream: %w", err)
}
}
return nil
}
func (s *Session) runAudio(ctx context.Context) (err error) {
for {
segment, err := s.audio.Next(ctx)
if err != nil {
return fmt.Errorf("failed to get next segment: %w", err)
}
if segment == nil {
return nil
}
err = s.writeSegment(ctx, segment)
if err != nil {
return fmt.Errorf("failed to write segment stream: %w", err)
}
}
}
func (s *Session) runVideo(ctx context.Context) (err error) {
for {
segment, err := s.video.Next(ctx)
if err != nil {
return fmt.Errorf("failed to get next segment: %w", err)
}
if segment == nil {
return nil
}
err = s.writeSegment(ctx, segment)
if err != nil {
return fmt.Errorf("failed to write segment stream: %w", err)
}
}
}
// Create a stream for an INIT segment and write the container.
func (s *Session) writeInit(ctx context.Context, init *MediaInit) (err error) {
temp, err := s.inner.OpenUniStreamSync(ctx)
if err != nil {
return fmt.Errorf("failed to create stream: %w", err)
}
if temp == nil {
// Not sure when this happens, perhaps when closing a connection?
return fmt.Errorf("received a nil stream from quic-go")
}
// Wrap the stream in an object that buffers writes instead of blocking.
stream := NewStream(temp)
s.streams.Add(stream.Run)
defer func() {
if err != nil {
stream.WriteCancel(1)
}
}()
stream.SetPriority(math.MaxInt)
err = stream.WriteMessage(Message{
Init: &MessageInit{Id: init.ID},
})
if err != nil {
return fmt.Errorf("failed to write init header: %w", err)
}
_, err = stream.Write(init.Raw)
if err != nil {
return fmt.Errorf("failed to write init data: %w", err)
}
err = stream.Close()
if err != nil {
return fmt.Errorf("failed to close init stream: %w", err)
}
return nil
}
// Create a stream for a segment and write the contents, chunk by chunk.
func (s *Session) writeSegment(ctx context.Context, segment *MediaSegment) (err error) {
temp, err := s.inner.OpenUniStreamSync(ctx)
if err != nil {
return fmt.Errorf("failed to create stream: %w", err)
}
if temp == nil {
// Not sure when this happens, perhaps when closing a connection?
return fmt.Errorf("received a nil stream from quic-go")
}
// Wrap the stream in an object that buffers writes instead of blocking.
stream := NewStream(temp)
s.streams.Add(stream.Run)
defer func() {
if err != nil {
stream.WriteCancel(1)
}
}()
ms := int(segment.timestamp / time.Millisecond)
// newer segments take priority
stream.SetPriority(ms)
err = stream.WriteMessage(Message{
Segment: &MessageSegment{
Init: segment.Init.ID,
Timestamp: ms,
},
})
if err != nil {
return fmt.Errorf("failed to write segment header: %w", err)
}
for {
// Get the next fragment
buf, err := segment.Read(ctx)
if errors.Is(err, io.EOF) {
break
} else if err != nil {
return fmt.Errorf("failed to read segment data: %w", err)
}
// NOTE: This won't block because of our wrapper
_, err = stream.Write(buf)
if err != nil {
return fmt.Errorf("failed to write segment data: %w", err)
}
}
err = stream.Close()
if err != nil {
return fmt.Errorf("failed to close segemnt stream: %w", err)
}
return nil
}
func (s *Session) setDebug(msg *MessageDebug) {
s.conn.SetMaxBandwidth(uint64(msg.MaxBitrate))
}

144
server/internal/warp/stream.go
View File

@ -1,144 +0,0 @@
package warp
import (
"context"
"encoding/binary"
"encoding/json"
"fmt"
"sync"
"github.com/kixelated/webtransport-go"
)
// Wrapper around quic.SendStream to make Write non-blocking.
// Otherwise we can't write to multiple concurrent streams in the same goroutine.
type Stream struct {
inner webtransport.SendStream
chunks [][]byte
closed bool
err error
notify chan struct{}
mutex sync.Mutex
}
func NewStream(inner webtransport.SendStream) (s *Stream) {
s = new(Stream)
s.inner = inner
s.notify = make(chan struct{})
return s
}
func (s *Stream) Run(ctx context.Context) (err error) {
defer func() {
s.mutex.Lock()
s.err = err
s.mutex.Unlock()
}()
for {
s.mutex.Lock()
chunks := s.chunks
notify := s.notify
closed := s.closed
s.chunks = s.chunks[len(s.chunks):]
s.mutex.Unlock()
for _, chunk := range chunks {
_, err = s.inner.Write(chunk)
if err != nil {
return err
}
}
if closed {
return s.inner.Close()
}
if len(chunks) == 0 {
select {
case <-ctx.Done():
return ctx.Err()
case <-notify:
}
}
}
}
func (s *Stream) Write(buf []byte) (n int, err error) {
s.mutex.Lock()
defer s.mutex.Unlock()
if s.err != nil {
return 0, s.err
}
if s.closed {
return 0, fmt.Errorf("closed")
}
// Make a copy of the buffer so it's long lived
buf = append([]byte{}, buf...)
s.chunks = append(s.chunks, buf)
// Wake up the writer
close(s.notify)
s.notify = make(chan struct{})
return len(buf), nil
}
func (s *Stream) WriteMessage(msg Message) (err error) {
payload, err := json.Marshal(msg)
if err != nil {
return fmt.Errorf("failed to marshal message: %w", err)
}
var size [4]byte
binary.BigEndian.PutUint32(size[:], uint32(len(payload)+8))
_, err = s.Write(size[:])
if err != nil {
return fmt.Errorf("failed to write size: %w", err)
}
_, err = s.Write([]byte("warp"))
if err != nil {
return fmt.Errorf("failed to write atom header: %w", err)
}
_, err = s.Write(payload)
if err != nil {
return fmt.Errorf("failed to write payload: %w", err)
}
return nil
}
func (s *Stream) WriteCancel(code webtransport.StreamErrorCode) {
s.inner.CancelWrite(code)
}
func (s *Stream) SetPriority(prio int) {
s.inner.SetPriority(prio)
}
func (s *Stream) Close() (err error) {
s.mutex.Lock()
defer s.mutex.Unlock()
if s.err != nil {
return s.err
}
s.closed = true
// Wake up the writer
close(s.notify)
s.notify = make(chan struct{})
return nil
}

56
server/main.go
View File

@ -1,56 +0,0 @@
package main
import (
"context"
"crypto/tls"
"flag"
"fmt"
"log"
"github.com/kixelated/invoker"
"github.com/kixelated/warp/server/internal/warp"
)
func main() {
err := run(context.Background())
if err != nil {
log.Fatal(err)
}
}
func run(ctx context.Context) (err error) {
addr := flag.String("addr", ":4443", "HTTPS server address")
cert := flag.String("tls-cert", "../cert/localhost.crt", "TLS certificate file path")
key := flag.String("tls-key", "../cert/localhost.key", "TLS certificate file path")
logDir := flag.String("log-dir", "", "logs will be written to the provided directory")
dash := flag.String("dash", "../media/playlist.mpd", "DASH playlist path")
flag.Parse()
media, err := warp.NewMedia(*dash)
if err != nil {
return fmt.Errorf("failed to open media: %w", err)
}
tlsCert, err := tls.LoadX509KeyPair(*cert, *key)
if err != nil {
return fmt.Errorf("failed to load TLS certificate: %w", err)
}
warpConfig := warp.Config{
Addr: *addr,
Cert: &tlsCert,
LogDir: *logDir,
Media: media,
}
warpServer, err := warp.New(warpConfig)
if err != nil {
return fmt.Errorf("failed to create warp server: %w", err)
}
log.Printf("listening on %s", *addr)
return invoker.Run(ctx, invoker.Interrupt, warpServer.Run)
}

3
server/src/lib.rs Normal file
View File

@ -0,0 +1,3 @@
pub mod media;
pub mod session;
pub mod transport;

38
server/src/main.rs Normal file
View File

@ -0,0 +1,38 @@
use warp::{session, transport};
use clap::Parser;
/// Search for a pattern in a file and display the lines that contain it.
#[derive(Parser)]
struct Cli {
/// Listen on this address
#[arg(short, long, default_value = "[::]:4443")]
addr: String,
/// Use the certificate file at this path
#[arg(short, long, default_value = "../cert/localhost.crt")]
cert: String,
/// Use the private key at this path
#[arg(short, long, default_value = "../cert/localhost.key")]
key: String,
/// Use the media file at this path
#[arg(short, long, default_value = "../media/fragmented.mp4")]
media: String,
}
fn main() -> anyhow::Result<()> {
env_logger::init();
let args = Cli::parse();
let server_config = transport::Config {
addr: args.addr,
cert: args.cert,
key: args.key,
};
let mut server = transport::Server::<session::Session>::new(server_config).unwrap();
server.run()
}

3
server/src/media/mod.rs Normal file
View File

@ -0,0 +1,3 @@
mod source;
pub use source::{Fragment, Source};

230
server/src/media/source.rs Normal file
View File

@ -0,0 +1,230 @@
use std::collections::{HashMap, VecDeque};
use std::io::Read;
use std::{fs, io, time};
use anyhow;
use mp4;
use mp4::ReadBox;
pub struct Source {
// We read the file once, in order, and don't seek backwards.
reader: io::BufReader<fs::File>,
// The timestamp when the broadcast "started", so we can sleep to simulate a live stream.
start: time::Instant,
// The initialization payload; ftyp + moov boxes.
pub init: Vec<u8>,
// The timescale used for each track.
timescales: HashMap<u32, u32>,
// Any fragments parsed and ready to be returned by next().
fragments: VecDeque<Fragment>,
}
pub struct Fragment {
// The track ID for the fragment.
pub track_id: u32,
// The data of the fragment.
pub data: Vec<u8>,
// Whether this fragment is a keyframe.
pub keyframe: bool,
// The timestamp of the fragment, in milliseconds, to simulate a live stream.
pub timestamp: u64,
}
impl Source {
pub fn new(path: &str) -> anyhow::Result<Self> {
let f = fs::File::open(path)?;
let mut reader = io::BufReader::new(f);
let start = time::Instant::now();
let ftyp = read_atom(&mut reader)?;
anyhow::ensure!(&ftyp[4..8] == b"ftyp", "expected ftyp atom");
let moov = read_atom(&mut reader)?;
anyhow::ensure!(&moov[4..8] == b"moov", "expected moov atom");
let mut init = ftyp;
init.extend(&moov);
// We're going to parse the moov box.
// We have to read the moov box header to correctly advance the cursor for the mp4 crate.
let mut moov_reader = io::Cursor::new(&moov);
let moov_header = mp4::BoxHeader::read(&mut moov_reader)?;
// Parse the moov box so we can detect the timescales for each track.
let moov = mp4::MoovBox::read_box(&mut moov_reader, moov_header.size)?;
Ok(Self {
reader,
start,
init,
timescales: timescales(&moov),
fragments: VecDeque::new(),
})
}
pub fn fragment(&mut self) -> anyhow::Result<Option<Fragment>> {
if self.fragments.is_empty() {
self.parse()?;
};
if self.timeout().is_some() {
return Ok(None);
}
Ok(self.fragments.pop_front())
}
fn parse(&mut self) -> anyhow::Result<()> {
loop {
let atom = read_atom(&mut self.reader)?;
let mut reader = io::Cursor::new(&atom);
let header = mp4::BoxHeader::read(&mut reader)?;
match header.name {
mp4::BoxType::FtypBox | mp4::BoxType::MoovBox => {
anyhow::bail!("must call init first")
}
mp4::BoxType::MoofBox => {
let moof = mp4::MoofBox::read_box(&mut reader, header.size)?;
if moof.trafs.len() != 1 {
// We can't split the mdat atom, so this is impossible to support
anyhow::bail!("multiple tracks per moof atom")
}
self.fragments.push_back(Fragment {
track_id: moof.trafs[0].tfhd.track_id,
data: atom,
keyframe: has_keyframe(&moof),
timestamp: first_timestamp(&moof).expect("couldn't find timestamp"),
})
}
mp4::BoxType::MdatBox => {
let moof = self.fragments.back().expect("no atom before mdat");
self.fragments.push_back(Fragment {
track_id: moof.track_id,
data: atom,
keyframe: false,
timestamp: moof.timestamp,
});
// We have some media data, return so we can start sending it.
return Ok(());
}
_ => {
// Skip unknown atoms
}
}
}
}
// Simulate a live stream by sleeping until the next timestamp in the media.
pub fn timeout(&self) -> Option<time::Duration> {
let next = self.fragments.front()?;
let timestamp = next.timestamp;
// Find the timescale for the track.
let timescale = self.timescales.get(&next.track_id).unwrap();
let delay = time::Duration::from_millis(1000 * timestamp / *timescale as u64);
let elapsed = self.start.elapsed();
delay.checked_sub(elapsed)
}
}
// Read a full MP4 atom into a vector.
pub fn read_atom<R: Read>(reader: &mut R) -> anyhow::Result<Vec<u8>> {
// Read the 8 bytes for the size + type
let mut buf = [0u8; 8];
reader.read_exact(&mut buf)?;
// Convert the first 4 bytes into the size.
let size = u32::from_be_bytes(buf[0..4].try_into()?) as u64;
//let typ = &buf[4..8].try_into().ok().unwrap();
let mut raw = buf.to_vec();
let mut limit = match size {
// Runs until the end of the file.
0 => reader.take(u64::MAX),
// The next 8 bytes are the extended size to be used instead.
1 => {
reader.read_exact(&mut buf)?;
let size_large = u64::from_be_bytes(buf);
anyhow::ensure!(
size_large >= 16,
"impossible extended box size: {}",
size_large
);
reader.take(size_large - 16)
}
2..=7 => {
anyhow::bail!("impossible box size: {}", size)
}
// Otherwise read based on the size.
size => reader.take(size - 8),
};
// Append to the vector and return it.
limit.read_to_end(&mut raw)?;
Ok(raw)
}
fn has_keyframe(moof: &mp4::MoofBox) -> bool {
for traf in &moof.trafs {
// TODO trak default flags if this is None
let default_flags = traf.tfhd.default_sample_flags.unwrap_or_default();
let trun = match &traf.trun {
Some(t) => t,
None => return false,
};
for i in 0..trun.sample_count {
let mut flags = match trun.sample_flags.get(i as usize) {
Some(f) => *f,
None => default_flags,
};
if i == 0 && trun.first_sample_flags.is_some() {
flags = trun.first_sample_flags.unwrap();
}
// https://chromium.googlesource.com/chromium/src/media/+/master/formats/mp4/track_run_iterator.cc#177
let keyframe = (flags >> 24) & 0x3 == 0x2; // kSampleDependsOnNoOther
let non_sync = (flags >> 16) & 0x1 == 0x1; // kSampleIsNonSyncSample
if keyframe && !non_sync {
return true;
}
}
}
false
}
fn first_timestamp(moof: &mp4::MoofBox) -> Option<u64> {
Some(moof.trafs.first()?.tfdt.as_ref()?.base_media_decode_time)
}
fn timescales(moov: &mp4::MoovBox) -> HashMap<u32, u32> {
moov.traks
.iter()
.map(|trak| (trak.tkhd.track_id, trak.mdia.mdhd.timescale))
.collect()
}

37
server/src/session/message.rs Normal file
View File

@ -0,0 +1,37 @@
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize)]
pub struct Message {
pub init: Option<Init>,
pub segment: Option<Segment>,
}
#[derive(Serialize, Deserialize)]
pub struct Init {}
#[derive(Serialize, Deserialize)]
pub struct Segment {
pub track_id: u32,
}
impl Message {
pub fn new() -> Self {
Message {
init: None,
segment: None,
}
}
pub fn serialize(&self) -> anyhow::Result<Vec<u8>> {
let str = serde_json::to_string(self)?;
let bytes = str.as_bytes();
let size = bytes.len() + 8;
let mut out = Vec::with_capacity(size);
out.extend_from_slice(&(size as u32).to_be_bytes());
out.extend_from_slice(b"warp");
out.extend_from_slice(bytes);
Ok(out)
}
}

154
server/src/session/mod.rs Normal file
View File

@ -0,0 +1,154 @@
mod message;
use std::collections::hash_map as hmap;
use std::time;
use quiche;
use quiche::h3::webtransport;
use crate::{media, transport};
#[derive(Default)]
pub struct Session {
media: Option<media::Source>,
streams: transport::Streams, // An easy way of buffering stream data.
tracks: hmap::HashMap<u32, u64>, // map from track_id to current stream_id
}
impl transport::App for Session {
// Process any updates to a session.
fn poll(
&mut self,
conn: &mut quiche::Connection,
session: &mut webtransport::ServerSession,
) -> anyhow::Result<()> {
loop {
let event = match session.poll(conn) {
Err(webtransport::Error::Done) => break,
Err(e) => return Err(e.into()),
Ok(e) => e,
};
log::debug!("webtransport event {:?}", event);
match event {
webtransport::ServerEvent::ConnectRequest(_req) => {
// you can handle request with
// req.authority()
// req.path()
// and you can validate this request with req.origin()
session.accept_connect_request(conn, None)?;
// TODO
let media = media::Source::new("../media/fragmented.mp4")?;
let init = &media.init;
// Create a JSON header.
let mut message = message::Message::new();
message.init = Some(message::Init {});
let data = message.serialize()?;
// Create a new stream and write the header.
let stream_id = session.open_stream(conn, false)?;
self.streams.send(conn, stream_id, data.as_slice(), false)?;
self.streams.send(conn, stream_id, init.as_slice(), true)?;
self.media = Some(media);
}
webtransport::ServerEvent::StreamData(stream_id) => {
let mut buf = vec![0; 10000];
while let Ok(len) = session.recv_stream_data(conn, stream_id, &mut buf) {
let _stream_data = &buf[0..len];
}
}
_ => {}
}
}
// Send any pending stream data.
// NOTE: This doesn't return an error because it's async, and would be confusing.
self.streams.poll(conn);
// Fetch the next media fragment, possibly queuing up stream data.
self.poll_source(conn, session)?;
Ok(())
}
fn timeout(&self) -> Option<time::Duration> {
self.media.as_ref().and_then(|m| m.timeout())
}
}
impl Session {
fn poll_source(
&mut self,
conn: &mut quiche::Connection,
session: &mut webtransport::ServerSession,
) -> anyhow::Result<()> {
// Get the media source once the connection is established.
let media = match &mut self.media {
Some(m) => m,
None => return Ok(()),
};
// Get the next media fragment.
let fragment = match media.fragment()? {
Some(f) => f,
None => return Ok(()),
};
let stream_id = match self.tracks.get(&fragment.track_id) {
// Close the old stream.
Some(stream_id) if fragment.keyframe => {
self.streams.send(conn, *stream_id, &[], true)?;
None
}
// Use the existing stream
Some(stream_id) => Some(*stream_id),
// No existing stream.
_ => None,
};
let stream_id = match stream_id {
// Use the existing stream,
Some(stream_id) => stream_id,
// Open a new stream.
None => {
// Create a new unidirectional stream.
let stream_id = session.open_stream(conn, false)?;
// Set the stream priority to be equal to the timestamp.
// We subtract from u64::MAX so newer media is sent important.
// TODO prioritize audio
let order = u64::MAX - fragment.timestamp;
self.streams.send_order(conn, stream_id, order);
// Encode a JSON header indicating this is a new track.
let mut message: message::Message = message::Message::new();
message.segment = Some(message::Segment {
track_id: fragment.track_id,
});
// Write the header.
let data = message.serialize()?;
self.streams.send(conn, stream_id, &data, false)?;
// Keep a mapping from the track id to the current stream id.
self.tracks.insert(fragment.track_id, stream_id);
stream_id
}
};
// Write the current fragment.
let data = fragment.data.as_slice();
self.streams.send(conn, stream_id, data, false)?;
Ok(())
}
}

12
server/src/transport/app.rs Normal file
View File

@ -0,0 +1,12 @@
use std::time;
use quiche::h3::webtransport;
pub trait App: Default {
fn poll(
&mut self,
conn: &mut quiche::Connection,
session: &mut webtransport::ServerSession,
) -> anyhow::Result<()>;
fn timeout(&self) -> Option<time::Duration>;
}

15
server/src/transport/connection.rs Normal file
View File

@ -0,0 +1,15 @@
use quiche;
use quiche::h3::webtransport;
use std::collections::hash_map as hmap;
pub type Id = quiche::ConnectionId<'static>;
use super::app;
pub type Map<T> = hmap::HashMap<Id, Connection<T>>;
pub struct Connection<T: app::App> {
pub quiche: quiche::Connection,
pub session: Option<webtransport::ServerSession>,
pub app: T,
}

8
server/src/transport/mod.rs Normal file
View File

@ -0,0 +1,8 @@
mod app;
mod connection;
mod server;
mod streams;
pub use app::App;
pub use server::{Config, Server};
pub use streams::Streams;

398
server/src/transport/server.rs Normal file
View File

@ -0,0 +1,398 @@
use std::io;
use quiche::h3::webtransport;
use super::app;
use super::connection;
const MAX_DATAGRAM_SIZE: usize = 1350;
pub struct Server<T: app::App> {
// IO stuff
socket: mio::net::UdpSocket,
poll: mio::Poll,
events: mio::Events,
// QUIC stuff
quic: quiche::Config,
seed: ring::hmac::Key, // connection ID seed
conns: connection::Map<T>,
}
pub struct Config {
pub addr: String,
pub cert: String,
pub key: String,
}
impl<T: app::App> Server<T> {
pub fn new(config: Config) -> io::Result<Self> {
// Listen on the provided socket address
let addr = config.addr.parse().unwrap();
let mut socket = mio::net::UdpSocket::bind(addr).unwrap();
// Setup the event loop.
let poll = mio::Poll::new().unwrap();
let events = mio::Events::with_capacity(1024);
poll.registry()
.register(&mut socket, mio::Token(0), mio::Interest::READABLE)
.unwrap();
// Generate random values for connection IDs.
let rng = ring::rand::SystemRandom::new();
let seed = ring::hmac::Key::generate(ring::hmac::HMAC_SHA256, &rng).unwrap();
// Create the configuration for the QUIC conns.
let mut quic = quiche::Config::new(quiche::PROTOCOL_VERSION).unwrap();
quic.load_cert_chain_from_pem_file(&config.cert).unwrap();
quic.load_priv_key_from_pem_file(&config.key).unwrap();
quic.set_application_protos(quiche::h3::APPLICATION_PROTOCOL)
.unwrap();
quic.set_max_idle_timeout(5000);
quic.set_max_recv_udp_payload_size(MAX_DATAGRAM_SIZE);
quic.set_max_send_udp_payload_size(MAX_DATAGRAM_SIZE);
quic.set_initial_max_data(10_000_000);
quic.set_initial_max_stream_data_bidi_local(1_000_000);
quic.set_initial_max_stream_data_bidi_remote(1_000_000);
quic.set_initial_max_stream_data_uni(1_000_000);
quic.set_initial_max_streams_bidi(100);
quic.set_initial_max_streams_uni(100);
quic.set_disable_active_migration(true);
quic.enable_early_data();
quic.enable_dgram(true, 65536, 65536);
let conns = Default::default();
Ok(Server {
socket,
poll,
events,
quic,
seed,
conns,
})
}
pub fn run(&mut self) -> anyhow::Result<()> {
log::info!("listening on {}", self.socket.local_addr()?);
loop {
self.wait()?;
self.receive()?;
self.app()?;
self.send()?;
self.cleanup();
}
}
pub fn wait(&mut self) -> anyhow::Result<()> {
// Find the shorter timeout from all the active connections.
//
// TODO: use event loop that properly supports timers
let timeout = self
.conns
.values()
.filter_map(|c| {
let timeout = c.quiche.timeout();
let expires = c.app.timeout();
match (timeout, expires) {
(Some(a), Some(b)) => Some(a.min(b)),
(Some(a), None) => Some(a),
(None, Some(b)) => Some(b),
(None, None) => None,
}
})
.min();
self.poll.poll(&mut self.events, timeout).unwrap();
// If the event loop reported no events, it means that the timeout
// has expired, so handle it without attempting to read packets. We
// will then proceed with the send loop.
if self.events.is_empty() {
for conn in self.conns.values_mut() {
conn.quiche.on_timeout();
}
}
Ok(())
}
// Reads packets from the socket, updating any internal connection state.
fn receive(&mut self) -> anyhow::Result<()> {
let mut src = [0; MAX_DATAGRAM_SIZE];
// Try reading any data currently available on the socket.
loop {
let (len, from) = match self.socket.recv_from(&mut src) {
Ok(v) => v,
Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => return Ok(()),
Err(e) => return Err(e.into()),
};
let src = &mut src[..len];
let info = quiche::RecvInfo {
to: self.socket.local_addr().unwrap(),
from,
};
// Parse the QUIC packet's header.
let hdr = quiche::Header::from_slice(src, quiche::MAX_CONN_ID_LEN).unwrap();
let conn_id = ring::hmac::sign(&self.seed, &hdr.dcid);
let conn_id = &conn_id.as_ref()[..quiche::MAX_CONN_ID_LEN];
let conn_id = conn_id.to_vec().into();
// Check if it's an existing connection.
if let Some(conn) = self.conns.get_mut(&hdr.dcid) {
conn.quiche.recv(src, info)?;
if conn.session.is_none() && conn.quiche.is_established() {
conn.session = Some(webtransport::ServerSession::with_transport(
&mut conn.quiche,
)?)
}
continue;
} else if let Some(conn) = self.conns.get_mut(&conn_id) {
conn.quiche.recv(src, info)?;
// TODO is this needed here?
if conn.session.is_none() && conn.quiche.is_established() {
conn.session = Some(webtransport::ServerSession::with_transport(
&mut conn.quiche,
)?)
}
continue;
}
if hdr.ty != quiche::Type::Initial {
log::warn!("unknown connection ID");
continue;
}
let mut dst = [0; MAX_DATAGRAM_SIZE];
if !quiche::version_is_supported(hdr.version) {
let len = quiche::negotiate_version(&hdr.scid, &hdr.dcid, &mut dst).unwrap();
let dst = &dst[..len];
self.socket.send_to(dst, from).unwrap();
continue;
}
let mut scid = [0; quiche::MAX_CONN_ID_LEN];
scid.copy_from_slice(&conn_id);
let scid = quiche::ConnectionId::from_ref(&scid);
// Token is always present in Initial packets.
let token = hdr.token.as_ref().unwrap();
// Do stateless retry if the client didn't send a token.
if token.is_empty() {
let new_token = mint_token(&hdr, &from);
let len = quiche::retry(
&hdr.scid,
&hdr.dcid,
&scid,
&new_token,
hdr.version,
&mut dst,
)
.unwrap();
let dst = &dst[..len];
self.socket.send_to(dst, from).unwrap();
continue;
}
let odcid = validate_token(&from, token);
// The token was not valid, meaning the retry failed, so
// drop the packet.
if odcid.is_none() {
log::warn!("invalid token");
continue;
}
if scid.len() != hdr.dcid.len() {
log::warn!("invalid connection ID");
continue;
}
// Reuse the source connection ID we sent in the Retry packet,
// instead of changing it again.
let conn_id = hdr.dcid.clone();
let local_addr = self.socket.local_addr().unwrap();
log::debug!("new connection: dcid={:?} scid={:?}", hdr.dcid, scid);
let mut conn =
quiche::accept(&conn_id, odcid.as_ref(), local_addr, from, &mut self.quic)?;
// Log each session with QLOG if the ENV var is set.
if let Some(dir) = std::env::var_os("QLOGDIR") {
let id = format!("{:?}", &scid);
let mut path = std::path::PathBuf::from(dir);
let filename = format!("server-{id}.sqlog");
path.push(filename);
let writer = match std::fs::File::create(&path) {
Ok(f) => std::io::BufWriter::new(f),
Err(e) => panic!(
"Error creating qlog file attempted path was {:?}: {}",
path, e
),
};
conn.set_qlog(
std::boxed::Box::new(writer),
"warp-server qlog".to_string(),
format!("{} id={}", "warp-server qlog", id),
);
}
// Process potentially coalesced packets.
conn.recv(src, info)?;
let user = connection::Connection {
quiche: conn,
session: None,
app: T::default(),
};
self.conns.insert(conn_id, user);
}
}
pub fn app(&mut self) -> anyhow::Result<()> {
for conn in self.conns.values_mut() {
if conn.quiche.is_closed() {
continue;
}
if let Some(session) = &mut conn.session {
if let Err(e) = conn.app.poll(&mut conn.quiche, session) {
log::debug!("app error: {:?}", e);
// Close the connection on any application error
let reason = format!("app error: {:?}", e);
conn.quiche.close(true, 0xff, reason.as_bytes()).ok();
}
}
}
Ok(())
}
// Generate outgoing QUIC packets for all active connections and send
// them on the UDP socket, until quiche reports that there are no more
// packets to be sent.
pub fn send(&mut self) -> anyhow::Result<()> {
for conn in self.conns.values_mut() {
let conn = &mut conn.quiche;
if let Err(e) = send_conn(&self.socket, conn) {
log::error!("{} send failed: {:?}", conn.trace_id(), e);
conn.close(false, 0x1, b"fail").ok();
}
}
Ok(())
}
pub fn cleanup(&mut self) {
// Garbage collect closed connections.
self.conns.retain(|_, ref mut c| !c.quiche.is_closed());
}
}
// Send any pending packets for the connection over the socket.
fn send_conn(socket: &mio::net::UdpSocket, conn: &mut quiche::Connection) -> anyhow::Result<()> {
let mut pkt = [0; MAX_DATAGRAM_SIZE];
loop {
let (size, info) = match conn.send(&mut pkt) {
Ok(v) => v,
Err(quiche::Error::Done) => return Ok(()),
Err(e) => return Err(e.into()),
};
let pkt = &pkt[..size];
match socket.send_to(pkt, info.to) {
Err(e) if e.kind() == io::ErrorKind::WouldBlock => return Ok(()),
Err(e) => return Err(e.into()),
Ok(_) => (),
}
}
}
/// Generate a stateless retry token.
///
/// The token includes the static string `"quiche"` followed by the IP address
/// of the client and by the original destination connection ID generated by the
/// client.
///
/// Note that this function is only an example and doesn't do any cryptographic
/// authenticate of the token. *It should not be used in production system*.
fn mint_token(hdr: &quiche::Header, src: &std::net::SocketAddr) -> Vec<u8> {
let mut token = Vec::new();
token.extend_from_slice(b"quiche");
let addr = match src.ip() {
std::net::IpAddr::V4(a) => a.octets().to_vec(),
std::net::IpAddr::V6(a) => a.octets().to_vec(),
};
token.extend_from_slice(&addr);
token.extend_from_slice(&hdr.dcid);
token
}
/// Validates a stateless retry token.
///
/// This checks that the ticket includes the `"quiche"` static string, and that
/// the client IP address matches the address stored in the ticket.
///
/// Note that this function is only an example and doesn't do any cryptographic
/// authenticate of the token. *It should not be used in production system*.
fn validate_token<'a>(
src: &std::net::SocketAddr,
token: &'a [u8],
) -> Option<quiche::ConnectionId<'a>> {
if token.len() < 6 {
return None;
}
if &token[..6] != b"quiche" {
return None;
}
let token = &token[6..];
let addr = match src.ip() {
std::net::IpAddr::V4(a) => a.octets().to_vec(),
std::net::IpAddr::V6(a) => a.octets().to_vec(),
};
if token.len() < addr.len() || &token[..addr.len()] != addr.as_slice() {
return None;
}
Some(quiche::ConnectionId::from_ref(&token[addr.len()..]))
}

149
server/src/transport/streams.rs Normal file
View File

@ -0,0 +1,149 @@
use std::collections::VecDeque;
use anyhow;
use quiche;
#[derive(Default)]
pub struct Streams {
ordered: Vec<Stream>,
}
struct Stream {
id: u64,
order: u64,
buffer: VecDeque<u8>,
fin: bool,
}
impl Streams {
// Write the data to the given stream, buffering it if needed.
pub fn send(
&mut self,
conn: &mut quiche::Connection,
id: u64,
buf: &[u8],
fin: bool,
) -> anyhow::Result<()> {
if buf.is_empty() && !fin {
return Ok(());
}
// Get the index of the stream, or add it to the list of streams.
let pos = self
.ordered
.iter()
.position(|s| s.id == id)
.unwrap_or_else(|| {
// Create a new stream
let stream = Stream {
id,
buffer: VecDeque::new(),
fin: false,
order: 0, // Default to highest priority until send_order is called.
};
self.insert(conn, stream)
});
let stream = &mut self.ordered[pos];
// Check if we've already closed the stream, just in case.
if stream.fin && !buf.is_empty() {
anyhow::bail!("stream is already finished");
}
// If there's no data buffered, try to write it immediately.
let size = if stream.buffer.is_empty() {
match conn.stream_send(id, buf, fin) {
Ok(size) => size,
Err(quiche::Error::Done) => 0,
Err(e) => anyhow::bail!(e),
}
} else {
0
};
if size < buf.len() {
// Short write, save the rest for later.
stream.buffer.extend(&buf[size..]);
}
stream.fin |= fin;
Ok(())
}
// Flush any pending stream data.
pub fn poll(&mut self, conn: &mut quiche::Connection) {
self.ordered.retain_mut(|s| s.poll(conn).is_ok());
}
// Set the send order of the stream.
pub fn send_order(&mut self, conn: &mut quiche::Connection, id: u64, order: u64) {
let mut stream = match self.ordered.iter().position(|s| s.id == id) {
// Remove the stream from the existing list.
Some(pos) => self.ordered.remove(pos),
// This is a new stream, insert it into the list.
None => Stream {
id,
buffer: VecDeque::new(),
fin: false,
order,
},
};
stream.order = order;
self.insert(conn, stream);
}
fn insert(&mut self, conn: &mut quiche::Connection, stream: Stream) -> usize {
// Look for the position to insert the stream.
let pos = match self
.ordered
.binary_search_by_key(&stream.order, |s| s.order)
{
Ok(pos) | Err(pos) => pos,
};
self.ordered.insert(pos, stream);
// Reprioritize all later streams.
// TODO we can avoid this if stream_priorty takes a u64
for (i, stream) in self.ordered[pos..].iter().enumerate() {
_ = conn.stream_priority(stream.id, (pos + i) as u8, true);
}
pos
}
}
impl Stream {
fn poll(&mut self, conn: &mut quiche::Connection) -> quiche::Result<()> {
// Keep reading from the buffer until it's empty.
while !self.buffer.is_empty() {
// VecDeque is a ring buffer, so we can't write the whole thing at once.
let parts = self.buffer.as_slices();
let size = conn.stream_send(self.id, parts.0, false)?;
if size == 0 {
// No more space available for this stream.
return Ok(());
}
// Remove the bytes that were written.
self.buffer.drain(..size);
}
if self.fin {
// Write the stream done signal.
conn.stream_send(self.id, &[], true)?;
Err(quiche::Error::Done)
} else {
Ok(())
}
}
}

4
player/.gitignore → web/.dockerignore
View File

@ -1,4 +1,4 @@
node_modules
.parcel-cache
dist
.parcel-cache
node_modules
fingerprint.hex

13
web/.eslintrc.cjs Normal file
View File

@ -0,0 +1,13 @@
/* eslint-env node */
module.exports = {
extends: ['eslint:recommended', 'plugin:@typescript-eslint/recommended'],
parser: '@typescript-eslint/parser',
plugins: ['@typescript-eslint'],
root: true,
ignorePatterns: [ 'dist', 'node_modules' ],
rules: {
"@typescript-eslint/ban-ts-comment": "off",
"@typescript-eslint/no-non-null-assertion": "off",
"@typescript-eslint/no-explicit-any": "off",
}
};

3
web/.gitignore vendored Normal file
View File

@ -0,0 +1,3 @@
node_modules
.parcel-cache
dist

0
player/.proxyrc.js → web/.proxyrc.js
View File

26
web/Dockerfile Normal file
View File

@ -0,0 +1,26 @@
# Use the official Node.js image as the build image
FROM node:latest
# Set the build directory
WORKDIR /build
# Copy the package.json and yarn.lock files to the container
COPY package*.json yarn.lock ./
# Install dependencies
RUN yarn install
# Copy the entire project to the container
COPY . .
# Expose port 4444 for serving the project
EXPOSE 4444
# Copy the certificate hash before running
VOLUME /cert
# Make a symlink to the certificate fingerprint
RUN ln -s /cert/localhost.hex fingerprint.hex
# Copy the certificate fingerprint and start the web server
CMD yarn parcel serve --https --cert /cert/localhost.crt --key /cert/localhost.key --port 4444

1
web/fingerprint.hex Symbolic link
View File

@ -0,0 +1 @@
../cert/localhost.hex

3
player/package.json → web/package.json
View File

@ -1,7 +1,8 @@
{
"license": "Apache-2.0",
"source": "src/index.html",
"scripts": {
"serve": "parcel serve --https --cert ../cert/localhost.crt --key ../cert/localhost.key --host localhost --port 4444 --open",
"serve": "parcel serve --https --cert ../cert/localhost.crt --key ../cert/localhost.key --port 4444 --open",
"build": "parcel build",
"check": "tsc --noEmit"
},

104
web/src/broadcaster/encoder.ts Normal file
View File

@ -0,0 +1,104 @@
import * as MP4 from "../mp4"
export class Encoder {
container: MP4.ISOFile
audio: AudioEncoder
video: VideoEncoder
constructor() {
this.container = new MP4.ISOFile();
this.audio = new AudioEncoder({
output: this.onAudio.bind(this),
error: console.warn,
});
this.video = new VideoEncoder({
output: this.onVideo.bind(this),
error: console.warn,
});
this.container.init();
this.audio.configure({
codec: "mp4a.40.2",
numberOfChannels: 2,
sampleRate: 44100,
// TODO bitrate
})
this.video.configure({
codec: "avc1.42002A", // TODO h.264 baseline
avc: { format: "avc" }, // or annexb
width: 1280,
height: 720,
// TODO bitrate
// TODO bitrateMode
// TODO framerate
// TODO latencyMode
})
}
onAudio(frame: EncodedAudioChunk, metadata: EncodedAudioChunkMetadata) {
const config = metadata.decoderConfig!
const track_id = 1;
if (!this.container.getTrackById(track_id)) {
this.container.addTrack({
id: track_id,
type: "mp4a", // TODO wrong
timescale: 1000, // TODO verify
channel_count: config.numberOfChannels,
samplerate: config.sampleRate,
description: config.description, // TODO verify
// TODO description_boxes?: Box[];
});
}
const buffer = new Uint8Array(frame.byteLength);
frame.copyTo(buffer);
// TODO cts?
const sample = this.container.addSample(track_id, buffer, {
is_sync: frame.type == "key",
duration: frame.duration!,
dts: frame.timestamp,
});
const stream = this.container.createSingleSampleMoof(sample);
}
onVideo(frame: EncodedVideoChunk, metadata?: EncodedVideoChunkMetadata) {
const config = metadata!.decoderConfig!
const track_id = 2;
if (!this.container.getTrackById(track_id)) {
this.container.addTrack({
id: 2,
type: "avc1",
width: config.codedWidth,
height: config.codedHeight,
timescale: 1000, // TODO verify
description: config.description, // TODO verify
// TODO description_boxes?: Box[];
});
}
const buffer = new Uint8Array(frame.byteLength);
frame.copyTo(buffer);
// TODO cts?
const sample = this.container.addSample(track_id, buffer, {
is_sync: frame.type == "key",
duration: frame.duration!,
dts: frame.timestamp,
});
const stream = this.container.createSingleSampleMoof(sample);
}
}

4
web/src/broadcaster/index.ts Normal file
View File

@ -0,0 +1,4 @@
export default class Broadcaster {
constructor() {
}
}

0
player/src/index.css → web/src/index.css
View File

5
player/src/index.html → web/src/index.html
View File

@ -2,7 +2,7 @@
<html>
<head>
<meta charset = "UTF-8">
<meta charset="UTF-8">
<title>WARP</title>
<link rel="stylesheet" href="index.css">
@ -11,7 +11,7 @@
<body>
<div id="player">
<div id="screen">
<div id="play"><span>click for audio</span></div>
<div id="play"><span>click to play</span></div>
<canvas id="video" width="1280" height="720"></canvas>
</div>
@ -31,4 +31,5 @@
<script src="index.ts" type="module"></script>
</body>
</html>

15
player/src/index.ts → web/src/index.ts
View File

@ -1,12 +1,13 @@
import Player from "./player"
import Transport from "./transport"
// @ts-ignore embed the certificate fingerprint using bundler
import fingerprintHex from 'bundle-text:../fingerprint.hex';
// Convert the hex to binary.
let fingerprint = [];
for (let c = 0; c < fingerprintHex.length-1; c += 2) {
fingerprint.push(parseInt(fingerprintHex.substring(c, c+2), 16));
for (let c = 0; c < fingerprintHex.length - 1; c += 2) {
fingerprint.push(parseInt(fingerprintHex.substring(c, c + 2), 16));
}
const params = new URLSearchParams(window.location.search)
@ -14,18 +15,22 @@ const params = new URLSearchParams(window.location.search)
const url = params.get("url") || "https://localhost:4443/watch"
const canvas = document.querySelector<HTMLCanvasElement>("canvas#video")!
const player = new Player({
const transport = new Transport({
url: url,
fingerprint: { // TODO remove when Chrome accepts the system CA
"algorithm": "sha-256",
"value": new Uint8Array(fingerprint),
},
canvas: canvas,
})
const player = new Player({
transport,
canvas: canvas.transferControlToOffscreen(),
})
const play = document.querySelector<HTMLElement>("#screen #play")!
let playFunc = (e: Event) => {
const playFunc = (e: Event) => {
player.play()
e.preventDefault()

5
player/src/mp4/index.ts → web/src/mp4/index.ts
View File

@ -4,7 +4,12 @@ export {
MP4File as File,
MP4ArrayBuffer as ArrayBuffer,
MP4Info as Info,
MP4Track as Track,
MP4AudioTrack as AudioTrack,
MP4VideoTrack as VideoTrack,
DataStream as Stream,
Box,
ISOFile,
Sample,
} from "mp4box"

14
player/src/mp4/init.ts → web/src/mp4/init.ts
View File

@ -20,19 +20,7 @@ export class InitParser {
// Create a promise that gets resolved once the init segment has been parsed.
this.info = new Promise((resolve, reject) => {
this.mp4box.onError = reject
// https://github.com/gpac/mp4box.js#onreadyinfo
this.mp4box.onReady = (info: MP4.Info) => {
if (!info.isFragmented) {
reject("expected a fragmented mp4")
}
if (info.tracks.length != 1) {
reject("expected a single track")
}
resolve(info)
}
this.mp4box.onReady = resolve
})
}

93
player/src/mp4/mp4box.d.ts → web/src/mp4/mp4box.d.ts vendored
View File

@ -1,7 +1,7 @@
// https://github.com/gpac/mp4box.js/issues/233
declare module "mp4box" {
interface MP4MediaTrack {
export interface MP4MediaTrack {
id: number;
created: Date;
modified: Date;
@ -19,26 +19,26 @@ declare module "mp4box" {
nb_samples: number;
}
interface MP4VideoData {
export interface MP4VideoData {
width: number;
height: number;
}
interface MP4VideoTrack extends MP4MediaTrack {
export interface MP4VideoTrack extends MP4MediaTrack {
video: MP4VideoData;
}
interface MP4AudioData {
export interface MP4AudioData {
sample_rate: number;
channel_count: number;
sample_size: number;
}
interface MP4AudioTrack extends MP4MediaTrack {
export interface MP4AudioTrack extends MP4MediaTrack {
audio: MP4AudioData;
}
type MP4Track = MP4VideoTrack | MP4AudioTrack;
export type MP4Track = MP4VideoTrack | MP4AudioTrack;
export interface MP4Info {
duration: number;
@ -82,7 +82,7 @@ declare module "mp4box" {
description: any;
data: ArrayBuffer;
size: number;
alreadyRead: number;
alreadyRead?: number;
duration: number;
cts: number;
dts: number;
@ -104,7 +104,7 @@ declare module "mp4box" {
const LITTLE_ENDIAN: boolean;
export class DataStream {
constructor(buffer: ArrayBuffer, byteOffset?: number, littleEndian?: boolean);
constructor(buffer?: ArrayBuffer, byteOffset?: number, littleEndian?: boolean);
getPosition(): number;
get byteLength(): number;
@ -144,5 +144,82 @@ declare module "mp4box" {
// TODO I got bored porting the remaining functions
}
export class Box {
write(stream: DataStream): void;
}
export interface TrackOptions {
id?: number;
type?: string;
width?: number;
height?: number;
duration?: number;
layer?: number;
timescale?: number;
media_duration?: number;
language?: string;
hdlr?: string;
// video
avcDecoderConfigRecord?: any;
// audio
balance?: number;
channel_count?: number;
samplesize?: number;
samplerate?: number;
//captions
namespace?: string;
schema_location?: string;
auxiliary_mime_types?: string;
description?: any;
description_boxes?: Box[];
default_sample_description_index_id?: number;
default_sample_duration?: number;
default_sample_size?: number;
default_sample_flags?: number;
}
export interface FileOptions {
brands?: string[];
timescale?: number;
rate?: number;
duration?: number;
width?: number;
}
export interface SampleOptions {
sample_description_index?: number;
duration?: number;
cts?: number;
dts?: number;
is_sync?: boolean;
is_leading?: number;
depends_on?: number;
is_depended_on?: number;
has_redundancy?: number;
degradation_priority?: number;
subsamples?: any;
}
// TODO add the remaining functions
// TODO move to another module
export class ISOFile {
constructor(stream?: DataStream);
init(options?: FileOptions): ISOFile;
addTrack(options?: TrackOptions): number;
addSample(track: number, data: ArrayBuffer, options?: SampleOptions): Sample;
createSingleSampleMoof(sample: Sample): Box;
// helpers
getTrackById(id: number): Box | undefined;
getTrexById(id: number): Box | undefined;
}
export { };
}

79
web/src/player/audio.ts Normal file
View File

@ -0,0 +1,79 @@
import * as Message from "./message";
import { Ring } from "./ring"
export default class Audio {
ring?: Ring;
queue: Array<AudioData>;
render?: number; // non-zero if requestAnimationFrame has been called
last?: number; // the timestamp of the last rendered frame, in microseconds
constructor(config: Message.Config) {
this.queue = []
}
push(frame: AudioData) {
// Drop any old frames
if (this.last && frame.timestamp <= this.last) {
frame.close()
return
}
// Insert the frame into the queue sorted by timestamp.
if (this.queue.length > 0 && this.queue[this.queue.length - 1].timestamp <= frame.timestamp) {
// Fast path because we normally append to the end.
this.queue.push(frame)
} else {
// Do a full binary search
let low = 0
let high = this.queue.length;
while (low < high) {
const mid = (low + high) >>> 1;
if (this.queue[mid].timestamp < frame.timestamp) low = mid + 1;
else high = mid;
}
this.queue.splice(low, 0, frame)
}
this.emit()
}
emit() {
const ring = this.ring
if (!ring) {
return
}
while (this.queue.length) {
let frame = this.queue[0];
if (ring.size() + frame.numberOfFrames > ring.capacity) {
// Buffer is full
break
}
const size = ring.write(frame)
if (size < frame.numberOfFrames) {
throw new Error("audio buffer is full")
}
this.last = frame.timestamp
frame.close()
this.queue.shift()
}
}
play(play: Message.Play) {
this.ring = new Ring(play.buffer)
if (!this.render) {
const sampleRate = 44100 // TODO dynamic
// Refresh every half buffer
const refresh = play.buffer.capacity / sampleRate * 1000 / 2
this.render = setInterval(this.emit.bind(this), refresh)
}
}
}

167
web/src/player/decoder.ts Normal file
View File

@ -0,0 +1,167 @@
import * as Message from "./message";
import * as MP4 from "../mp4"
import * as Stream from "../stream"
import Renderer from "./renderer"
export default class Decoder {
init: MP4.InitParser;
decoders: Map<number, AudioDecoder | VideoDecoder>;
renderer: Renderer;
constructor(renderer: Renderer) {
this.init = new MP4.InitParser();
this.decoders = new Map();
this.renderer = renderer;
}
async receiveInit(msg: Message.Init) {
let stream = new Stream.Reader(msg.reader, msg.buffer);
while (1) {
const data = await stream.read()
if (!data) break
this.init.push(data)
}
// TODO make sure the init segment is fully received
}
async receiveSegment(msg: Message.Segment) {
// Wait for the init segment to be fully received and parsed
const info = await this.init.info
const input = MP4.New();
input.onSamples = this.onSamples.bind(this);
input.onReady = (info: any) => {
// Extract all of the tracks, because we don't know if it's audio or video.
for (let track of info.tracks) {
input.setExtractionOptions(track.id, track, { nbSamples: 1 });
}
input.start();
}
// MP4box requires us to reparse the init segment unfortunately
let offset = 0;
for (let raw of this.init.raw) {
raw.fileStart = offset
offset = input.appendBuffer(raw)
}
const stream = new Stream.Reader(msg.reader, msg.buffer)
// For whatever reason, mp4box doesn't work until you read an atom at a time.
while (!await stream.done()) {
const raw = await stream.peek(4)
// TODO this doesn't support when size = 0 (until EOF) or size = 1 (extended size)
const size = new DataView(raw.buffer, raw.byteOffset, raw.byteLength).getUint32(0)
const atom = await stream.bytes(size)
// Make a copy of the atom because mp4box only accepts an ArrayBuffer unfortunately
let box = new Uint8Array(atom.byteLength);
box.set(atom)
// and for some reason we need to modify the underlying ArrayBuffer with offset
let buffer = box.buffer as MP4.ArrayBuffer
buffer.fileStart = offset
// Parse the data
offset = input.appendBuffer(buffer)
input.flush()
}
}
onSamples(track_id: number, track: MP4.Track, samples: MP4.Sample[]) {
let decoder = this.decoders.get(track_id);
if (!decoder) {
// We need a sample to initalize the video decoder, because of mp4box limitations.
let sample = samples[0];
if (isVideoTrack(track)) {
// Configure the decoder using the AVC box for H.264
// TODO it should be easy to support other codecs, just need to know the right boxes.
const avcc = sample.description.avcC;
if (!avcc) throw new Error("TODO only h264 is supported");
const description = new MP4.Stream(new Uint8Array(avcc.size), 0, false)
avcc.write(description)
const videoDecoder = new VideoDecoder({
output: this.renderer.push.bind(this.renderer),
error: console.warn,
});
videoDecoder.configure({
codec: track.codec,
codedHeight: track.video.height,
codedWidth: track.video.width,
description: description.buffer?.slice(8),
// optimizeForLatency: true
})
decoder = videoDecoder
} else if (isAudioTrack(track)) {
const audioDecoder = new AudioDecoder({
output: this.renderer.push.bind(this.renderer),
error: console.warn,
});
audioDecoder.configure({
codec: track.codec,
numberOfChannels: track.audio.channel_count,
sampleRate: track.audio.sample_rate,
})
decoder = audioDecoder
} else {
throw new Error("unknown track type")
}
this.decoders.set(track_id, decoder)
}
for (let sample of samples) {
// Convert to microseconds
const timestamp = 1000 * 1000 * sample.dts / sample.timescale
const duration = 1000 * 1000 * sample.duration / sample.timescale
if (isAudioDecoder(decoder)) {
decoder.decode(new EncodedAudioChunk({
type: sample.is_sync ? "key" : "delta",
data: sample.data,
duration: duration,
timestamp: timestamp,
}))
} else if (isVideoDecoder(decoder)) {
decoder.decode(new EncodedVideoChunk({
type: sample.is_sync ? "key" : "delta",
data: sample.data,
duration: duration,
timestamp: timestamp,
}))
} else {
throw new Error("unknown decoder type")
}
}
}
}
function isAudioDecoder(decoder: AudioDecoder | VideoDecoder): decoder is AudioDecoder {
return decoder instanceof AudioDecoder
}
function isVideoDecoder(decoder: AudioDecoder | VideoDecoder): decoder is VideoDecoder {
return decoder instanceof VideoDecoder
}
function isAudioTrack(track: MP4.Track): track is MP4.AudioTrack {
return (track as MP4.AudioTrack).audio !== undefined;
}
function isVideoTrack(track: MP4.Track): track is MP4.VideoTrack {
return (track as MP4.VideoTrack).video !== undefined;
}

59
player/src/audio/index.ts → web/src/player/index.ts
View File

@ -1,45 +1,51 @@
import * as Message from "./message"
import Renderer from "./renderer"
import Decoder from "./decoder"
import * as Ring from "./ring"
import Transport from "../transport"
import { RingInit } from "./ring"
export interface Config {
transport: Transport
canvas: OffscreenCanvas;
}
// Abstracts the Worker and Worklet into a simpler API
// This class must be created on the main thread due to AudioContext.
export default class Audio {
export default class Player {
context: AudioContext;
worker: Worker;
worklet: Promise<AudioWorkletNode>;
constructor() {
// Assume 44.1kHz and two audio channels
const config = {
sampleRate: 44100,
ring: new RingInit(2, 4410), // 100ms at 44.1khz
}
transport: Transport
constructor(config: Config) {
this.transport = config.transport
this.transport.callback = this;
this.context = new AudioContext({
latencyHint: "interactive",
sampleRate: config.sampleRate,
sampleRate: 44100,
})
this.worker = this.setupWorker(config)
this.worklet = this.setupWorklet(config)
}
private setupWorker(config: Message.Config): Worker {
private setupWorker(config: Config): Worker {
const url = new URL('worker.ts', import.meta.url)
const worker = new Worker(url, {
name: "audio",
type: "module",
name: "media",
})
worker.postMessage({ config })
const msg = {
canvas: config.canvas,
}
worker.postMessage({ config: msg }, [msg.canvas])
return worker
}
private async setupWorklet(config: Message.Config): Promise<AudioWorkletNode> {
private async setupWorklet(config: Config): Promise<AudioWorkletNode> {
// Load the worklet source code.
const url = new URL('worklet.ts', import.meta.url)
await this.context.audioWorklet.addModule(url)
@ -53,8 +59,6 @@ export default class Audio {
console.error("Audio worklet error:", e)
};
worklet.port.postMessage({ config })
// Connect the worklet to the volume node and then to the speakers
worklet.connect(volume)
volume.connect(this.context.destination)
@ -62,16 +66,23 @@ export default class Audio {
return worklet
}
init(init: Message.Init) {
this.worker.postMessage({ init })
onInit(init: Message.Init) {
this.worker.postMessage({ init }, [init.buffer.buffer, init.reader])
}
segment(segment: Message.Segment) {
this.worker.postMessage({ segment }, [ segment.buffer.buffer, segment.reader ])
onSegment(segment: Message.Segment) {
this.worker.postMessage({ segment }, [segment.buffer.buffer, segment.reader])
}
play(play: Message.Play) {
async play() {
this.context.resume()
//this.worker.postMessage({ play })
const play = {
buffer: new Ring.Buffer(2, 44100 / 10), // 100ms of audio
}
const worklet = await this.worklet;
worklet.port.postMessage({ play })
this.worker.postMessage({ play })
}
}

21
web/src/player/message.ts Normal file
View File

@ -0,0 +1,21 @@
import * as Ring from "./ring"
export interface Config {
// video stuff
canvas: OffscreenCanvas;
}
export interface Init {
buffer: Uint8Array; // unread buffered data
reader: ReadableStream; // unread unbuffered data
}
export interface Segment {
buffer: Uint8Array; // unread buffered data
reader: ReadableStream; // unread unbuffered data
}
export interface Play {
timestamp?: number;
buffer: Ring.Buffer;
}

36
web/src/player/renderer.ts Normal file
View File

@ -0,0 +1,36 @@
import * as Message from "./message";
import Audio from "./audio"
import Video from "./video"
export default class Renderer {
audio: Audio;
video: Video;
constructor(config: Message.Config) {
this.audio = new Audio(config);
this.video = new Video(config);
}
push(frame: AudioData | VideoFrame) {
if (isAudioData(frame)) {
this.audio.push(frame);
} else if (isVideoFrame(frame)) {
this.video.push(frame);
} else {
throw new Error("unknown frame type")
}
}
play(play: Message.Play) {
this.audio.play(play);
this.video.play(play);
}
}
function isAudioData(frame: AudioData | VideoFrame): frame is AudioData {
return frame instanceof AudioData
}
function isVideoFrame(frame: AudioData | VideoFrame): frame is VideoFrame {
return frame instanceof VideoFrame
}

155
web/src/player/ring.ts Normal file
View File

@ -0,0 +1,155 @@
// Ring buffer with audio samples.
enum STATE {
READ_POS = 0, // The current read position
WRITE_POS, // The current write position
LENGTH // Clever way of saving the total number of enums values.
}
// No prototype to make this easier to send via postMessage
export class Buffer {
state: SharedArrayBuffer;
channels: SharedArrayBuffer[];
capacity: number;
constructor(channels: number, capacity: number) {
// Store the current state in a separate ring buffer.
this.state = new SharedArrayBuffer(STATE.LENGTH * Int32Array.BYTES_PER_ELEMENT)
// Create a buffer for each audio channel
this.channels = []
for (let i = 0; i < channels; i += 1) {
const buffer = new SharedArrayBuffer(capacity * Float32Array.BYTES_PER_ELEMENT)
this.channels.push(buffer)
}
this.capacity = capacity
}
}
export class Ring {
state: Int32Array;
channels: Float32Array[];
capacity: number;
constructor(buffer: Buffer) {
this.state = new Int32Array(buffer.state)
this.channels = []
for (let channel of buffer.channels) {
this.channels.push(new Float32Array(channel))
}
this.capacity = buffer.capacity
}
// Write samples for single audio frame, returning the total number written.
write(frame: AudioData): number {
let readPos = Atomics.load(this.state, STATE.READ_POS)
let writePos = Atomics.load(this.state, STATE.WRITE_POS)
const startPos = writePos
let endPos = writePos + frame.numberOfFrames;
if (endPos > readPos + this.capacity) {
endPos = readPos + this.capacity
if (endPos <= startPos) {
// No space to write
return 0
}
}
let startIndex = startPos % this.capacity;
let endIndex = endPos % this.capacity;
// Loop over each channel
for (let i = 0; i < this.channels.length; i += 1) {
const channel = this.channels[i]
if (startIndex < endIndex) {
// One continuous range to copy.
const full = channel.subarray(startIndex, endIndex)
frame.copyTo(full, {
planeIndex: i,
frameCount: endIndex - startIndex,
})
} else {
const first = channel.subarray(startIndex)
const second = channel.subarray(0, endIndex)
frame.copyTo(first, {
planeIndex: i,
frameCount: first.length,
})
// We need this conditional when startIndex == 0 and endIndex == 0
// When capacity=4410 and frameCount=1024, this was happening 52s into the audio.
if (second.length) {
frame.copyTo(second, {
planeIndex: i,
frameOffset: first.length,
frameCount: second.length,
})
}
}
}
Atomics.store(this.state, STATE.WRITE_POS, endPos)
return endPos - startPos
}
read(dst: Float32Array[]): number {
let readPos = Atomics.load(this.state, STATE.READ_POS)
let writePos = Atomics.load(this.state, STATE.WRITE_POS)
let startPos = readPos;
let endPos = startPos + dst[0].length;
if (endPos > writePos) {
endPos = writePos
if (endPos <= startPos) {
// Nothing to read
return 0
}
}
let startIndex = startPos % this.capacity;
let endIndex = endPos % this.capacity;
// Loop over each channel
for (let i = 0; i < dst.length; i += 1) {
if (i >= this.channels.length) {
// ignore excess channels
}
const input = this.channels[i]
const output = dst[i]
if (startIndex < endIndex) {
const full = input.subarray(startIndex, endIndex)
output.set(full)
} else {
const first = input.subarray(startIndex)
const second = input.subarray(0, endIndex)
output.set(first)
output.set(second, first.length)
}
}
Atomics.store(this.state, STATE.READ_POS, endPos)
return endPos - startPos
}
size() {
// TODO is this thread safe?
let readPos = Atomics.load(this.state, STATE.READ_POS)
let writePos = Atomics.load(this.state, STATE.WRITE_POS)
return writePos - readPos
}
}

75
player/src/video/renderer.ts → web/src/player/video.ts
View File

@ -1,24 +1,21 @@
import * as Message from "./message";
export default class Renderer {
export default class Video {
canvas: OffscreenCanvas;
queue: Array<VideoFrame>;
render: number; // non-zero if requestAnimationFrame has been called
sync?: DOMHighResTimeStamp; // the wall clock value for timestamp 0
last?: number; // the timestamp of the last rendered frame
sync?: number; // the wall clock value for timestamp 0, in microseconds
last?: number; // the timestamp of the last rendered frame, in microseconds
constructor(config: Message.Config) {
this.canvas = config.canvas;
this.queue = [];
this.render = 0;
}
emit(frame: VideoFrame) {
if (!this.sync) {
// Save the frame as the sync point
this.sync = performance.now() - frame.timestamp
}
push(frame: VideoFrame) {
// Drop any old frames
if (this.last && frame.timestamp <= this.last) {
frame.close()
@ -26,7 +23,7 @@ export default class Renderer {
}
// Insert the frame into the queue sorted by timestamp.
if (this.queue.length > 0 && this.queue[this.queue.length-1].timestamp <= frame.timestamp) {
if (this.queue.length > 0 && this.queue[this.queue.length - 1].timestamp <= frame.timestamp) {
// Fast path because we normally append to the end.
this.queue.push(frame)
} else {
@ -35,44 +32,54 @@ export default class Renderer {
let high = this.queue.length;
while (low < high) {
var mid = (low + high) >>> 1;
const mid = (low + high) >>> 1;
if (this.queue[mid].timestamp < frame.timestamp) low = mid + 1;
else high = mid;
}
this.queue.splice(low, 0, frame)
}
// Queue up to render the next frame.
if (!this.render) {
this.render = self.requestAnimationFrame(this.draw.bind(this))
}
}
draw(now: DOMHighResTimeStamp) {
// Determine the target timestamp.
const target = now - this.sync!
draw(now: number) {
// Draw and then queue up the next draw call.
this.drawOnce(now);
let frame = this.queue[0]
if (frame.timestamp >= target) {
// nothing to render yet, wait for the next animation frame
this.render = self.requestAnimationFrame(this.draw.bind(this))
// Queue up the new draw frame.
this.render = self.requestAnimationFrame(this.draw.bind(this))
}
drawOnce(now: number) {
// Convert to microseconds
now *= 1000;
if (!this.queue.length) {
return
}
this.queue.shift()
let frame = this.queue[0];
if (!this.sync) {
this.sync = now - frame.timestamp;
}
// Determine the target timestamp.
const target = now - this.sync
if (frame.timestamp >= target) {
// nothing to render yet, wait for the next animation frame
return
}
this.queue.shift();
// Check if we should skip some frames
while (this.queue.length) {
const next = this.queue[0]
if (next.timestamp > target) {
break
}
if (next.timestamp > target) break
frame.close()
this.queue.shift()
frame = next
frame = this.queue.shift()!;
}
const ctx = this.canvas.getContext("2d");
@ -80,12 +87,12 @@ export default class Renderer {
this.last = frame.timestamp;
frame.close()
}
if (this.queue.length > 0) {
play(play: Message.Play) {
// Queue up to render the next frame.
if (!this.render) {
this.render = self.requestAnimationFrame(this.draw.bind(this))
} else {
// Break the loop for now
this.render = 0
}
}
}

7
player/src/video/worker.ts → web/src/player/worker.ts
View File

@ -13,10 +13,13 @@ self.addEventListener('message', async (e: MessageEvent) => {
decoder = new Decoder(renderer)
} else if (e.data.init) {
const init = e.data.init as Message.Init
await decoder.init(init)
await decoder.receiveInit(init)
} else if (e.data.segment) {
const segment = e.data.segment as Message.Segment
await decoder.decode(segment)
await decoder.receiveSegment(segment)
} else if (e.data.play) {
const play = e.data.play as Message.Play
await renderer.play(play)
}
})

16
player/src/audio/worklet.ts → web/src/player/worklet.ts
View File

@ -19,19 +19,19 @@ class Renderer extends AudioWorkletProcessor {
}
onMessage(e: MessageEvent) {
if (e.data.config) {
this.config(e.data.config)
if (e.data.play) {
this.onPlay(e.data.play)
}
}
config(config: Message.Config) {
this.ring = new Ring(config.ring)
onPlay(play: Message.Play) {
this.ring = new Ring(play.buffer)
}
// Inputs and outputs in groups of 128 samples.
process(inputs: Float32Array[][], outputs: Float32Array[][], parameters: Record<string, Float32Array>): boolean {
if (!this.ring) {
// Not initialized yet
// Paused
return true
}
@ -40,7 +40,11 @@ class Renderer extends AudioWorkletProcessor {
}
const output = outputs[0]
this.ring.read(output)
const size = this.ring.read(output)
if (size < output.length) {
// TODO trigger rebuffering event
}
return true;
}

0
player/src/stream/index.ts → web/src/stream/index.ts
View File

0
player/src/stream/reader.ts → web/src/stream/reader.ts
View File

0
player/src/stream/writer.ts → web/src/stream/writer.ts
View File

96
web/src/transport/index.ts Normal file
View File

@ -0,0 +1,96 @@
import * as Stream from "../stream"
import * as Interface from "./interface"
export interface Config {
url: string;
fingerprint?: WebTransportHash; // the certificate fingerprint, temporarily needed for local development
}
export default class Transport {
quic: Promise<WebTransport>;
api: Promise<WritableStream>;
callback?: Interface.Callback;
constructor(config: Config) {
this.quic = this.connect(config)
// Create a unidirectional stream for all of our messages
this.api = this.quic.then((q) => {
return q.createUnidirectionalStream()
})
// async functions
this.receiveStreams()
}
async close() {
(await this.quic).close()
}
// Helper function to make creating a promise easier
private async connect(config: Config): Promise<WebTransport> {
let options: WebTransportOptions = {};
if (config.fingerprint) {
options.serverCertificateHashes = [ config.fingerprint ]
}
const quic = new WebTransport(config.url, options)
await quic.ready
return quic
}
async sendMessage(msg: any) {
const payload = JSON.stringify(msg)
const size = payload.length + 8
const stream = await this.api
const writer = new Stream.Writer(stream)
await writer.uint32(size)
await writer.string("warp")
await writer.string(payload)
writer.release()
}
async receiveStreams() {
const q = await this.quic
const streams = q.incomingUnidirectionalStreams.getReader()
for (;;) {
const result = await streams.read()
if (result.done) break
const stream = result.value
this.handleStream(stream) // don't await
}
}
async handleStream(stream: ReadableStream) {
const r = new Stream.Reader(stream)
while (!await r.done()) {
const size = await r.uint32();
const typ = new TextDecoder('utf-8').decode(await r.bytes(4));
if (typ != "warp") throw "expected warp atom"
if (size < 8) throw "atom too small"
const payload = new TextDecoder('utf-8').decode(await r.bytes(size - 8));
const msg = JSON.parse(payload)
if (msg.init) {
return this.callback?.onInit({
buffer: r.buffer,
reader: r.reader,
})
} else if (msg.segment) {
return this.callback?.onSegment({
buffer: r.buffer,
reader: r.reader,
})
} else {
console.warn("unknown message", msg);
}
}
}
}

14
web/src/transport/interface.ts Normal file
View File

@ -0,0 +1,14 @@
export interface Callback {
onInit(init: Init): any
onSegment(segment: Segment): any
}
export interface Init {
buffer: Uint8Array; // unread buffered data
reader: ReadableStream; // unread unbuffered data
}
export interface Segment {
buffer: Uint8Array; // unread buffered data
reader: ReadableStream; // unread unbuffered data
}

6
web/src/transport/message.ts Normal file
View File

@ -0,0 +1,6 @@
export interface Init {}
export interface Segment {}
export interface Debug {
max_bitrate: number
}

0
player/src/transport/webtransport.d.ts → web/src/transport/webtransport.d.ts vendored
View File

0
player/src/util/deferred.ts → web/src/util/deferred.ts
View File

0
player/src/util/index.ts → web/src/util/index.ts
View File

0
player/tsconfig.json → web/tsconfig.json
View File

0
player/yarn.lock → web/yarn.lock
View File