Tracking Http Upload Progress Using Golang

I wrote a simple cli tool to upload files to my Cloudreve instance. It’s easy to figure out which api to call when uploading files. However, it will be helpful to know how the upload is going.

Visualizing Speed

When creating an upload request, golang allows us to use a io.Reader as the request body. The body will be responsible for displaying upload progress.

type ProgressReader struct {
	Reader   io.Reader
	Progress atomic.Int64
	Size     int64
	Name     string
	Once     bool
	Done     atomic.Bool
}

func (progressReader *ProgressReader) NewLoop() {
	ticker := time.NewTicker(time.Second)
	var op int64
	for range ticker.C {
		p := progressReader.Progress.Load()
		KB := (p - op) / 1024
		var percent int64
		if progressReader.Size != 0 {
			percent = p * 100 / progressReader.Size
		} else {
			percent = 100
		}
		if KB < 1024 {
			fmt.Printf("%s: %dKB/s %d%%\n", progressReader.Name, KB, percent)
		} else {
			fmt.Printf("%s: %.2fMB/s %d%%\n", progressReader.Name, float64(KB)/1024, percent)
		}
		
		if progressReader.Done.Load() {
			ticker.Stop()
			return
		}
	}
}

func (progressReader *ProgressReader) Read(p []byte) (int, error) {
	n, err := progressReader.Reader.Read(p)
	progressReader.Progress.Add(int64(n))
	if !progressReader.Once {
		progressReader.Once = true
		go progressReader.NewLoop()
	}
	if err != nil {
		progressReader.Done.Store(true)
	}
	return n, err
}

func (progressReader *ProgressReader) Close() error {
	progressReader.Done.Store(true)
	return nil
}

This type will hold all the information we need: the name, the total size, current progress, whether the monitor goroutine has started and if the monitor goroutine should exit.

Restarting Slow Uploads

In practice, sometimes uploads become unreasonably slow. When this happens, I usually Ctrl + Z the upload process, then after some time resume it. The upload is interrupted, but because I have retry logic in case of error, the upload will be retried and usually the problem will go away.

Now that we know how to track the upload speed, we just need to set a slow threshold and if the speed is below the threshold for long enough, we cancel the request and retry it.

We could return an error as the body is being read. However, as golang will first buffer the body, then write it on the wire, it will have no use when the body is completely read.

Per golang documents, instead we should supply a context.Context in this case to control the entire lifetime of the request. This even works if the client has buffered all the body but failed to get a response in a timely manner.

It is very easy to adapt the above example to achieve the new requirements. It’s left as an exercise.