note

This article was last updated on August 9, 2023, 1 year ago. The content may be out of date.

Previous post shows how to create a mpd manifest from the result of yt-dlp and play it using various players. However, there are some problems when we use the manifest with other tools such as ffmpeg and other video players rather than shaka-player and dash.js, mainly in the form of slow buffering.

Determine the Cause of Slow Buffering

First we determine how a browser requests YouTube assets. Open devtools:

Request Query Parameters and Payload

We can find the request is a POST request with query parameter range and payload x\n. After some testing it can be determined that range parameter controls which part of the assets the browser is requesting just like http range requests.

And from yt-dlp, it seems YouTube will throttle requests without a small enough range.

Because we are using a reverse proxy, we can modify the request to add the range query parameter to avoid throttling.

Parsing and Modifying Requests

If the request doesn’t have a Range header or the range is too large, we forward the request using several requests with smaller range query attached, and write the responses to the client.

But first we need to know the size of the assets.

The results of yt-dlp actually have the file size of the assets. We just didn’t use it before.

type ydlJson struct {
	Duration         int `json:"duration"`
	RequestedFormats []struct {
		FormatID string `json:"format_id"`
		Url      string `json:"url"`
		Filesize int64  `json:"filesize"`
	} `json:"requested_formats"`
}

When proxying the requests, we also need to parse the Range header if there is any:

func parseRange(ranges string, size int64) (int64, int64, bool) {
	if ranges == "" {
		return 0, size - 1, true
	}

	bytesRange, ok := strings.CutPrefix(ranges, "bytes=")
	if ok {
		var (
			before string
			after  string
		)
		before, after, ok = strings.Cut(bytesRange, "-")
		if !ok {
			return 0, 0, false
		}

		start, err := strconv.ParseInt(before, 10, 64)
		if err != nil {
			return 0, 0, false
		}

		var end int64
		if after == "" {
			end = size - 1
		} else {
			end, err = strconv.ParseInt(after, 10, 64)
		}

		if err == nil {
			return start, end, true
		}
	}
	return 0, 0, false
}

This function only parses a single part range and treats an empty range as a range covering the whole of the content. Other types of ranges are not parsed. Using the start and the end indices, we can rewrite the requests into several requests.

var (
    ranges         = request.Header.Get("Range")
    start, end, ok = parseRange(ranges, size)
)
if ok && start <= end && end < size {
    var (
        headerWritten bool
        oldQuery      = req.URL.RawQuery
        seeker        = strings.NewReader("x\n")
    )
    req.Header.Del("Range")
    req.Method = http.MethodPost
    req.ContentLength = 2
    req.Body = io.NopCloser(seeker)
    for start <= end {
        nextStart := start + bodyLimit - 1
        if nextStart > end {
            nextStart = end
        }
        req.URL.RawQuery = fmt.Sprintf("%s&range=%d-%d", oldQuery, start, nextStart)
        _, _ = seeker.Seek(0, io.SeekStart)

        var resp *http.Response
        resp, err = http.DefaultClient.Do(req)
        if err != nil {
            if !headerWritten {
                http.Error(writer, http.StatusText(http.StatusBadGateway), http.StatusBadGateway)
            }
            return
        }

        if !headerWritten {
            header := writer.Header()
            copyHeader(header, resp.Header)
            header.Set("Content-Length", strconv.FormatInt(end-start+1, 10))
            if ranges == "" {
                writer.WriteHeader(http.StatusOK)
            } else {
                header.Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", start, end, size))
                writer.WriteHeader(http.StatusPartialContent)
            }
            headerWritten = true
        }

        _, err = io.Copy(writer, resp.Body)
        _ = resp.Body.Close()
        if err != nil {
            return
        }
        start = nextStart + 1
    }
    return
}