note

This article was last updated on December 10, 2023, 2 months ago. The content may be out of date.

I wrote a simple cli tool to upload files to my Cloudreve instance. It’s easy to figure out which api to call when uploading files. However, it will be helpful to know how the upload is going.

Visualizing Speed

When creating an upload request, golang allows us to use a io.Reader as the request body. The body will be responsible for displaying upload progress.

info

Although Cloudreve and its storage providers require Content-Length header to be set, thus requiring the total length to be known beforehand, this method is applicable even if the total is unknown. We lose out on the ability to track the percentage of the upload in this case though 🤷.

type ProgressReader struct {
	Reader   io.Reader
	Progress atomic.Int64
	Size     int64
	Name     string
	Once     bool
	Done     atomic.Bool
}

func (progressReader *ProgressReader) NewLoop() {
	ticker := time.NewTicker(time.Second)
	var op int64
	for range ticker.C {
		p := progressReader.Progress.Load()
		KB := (p - op) / 1024
		var percent int64
		if progressReader.Size != 0 {
			percent = p * 100 / progressReader.Size
		} else {
			percent = 100
		}
		if KB < 1024 {
			fmt.Printf("%s: %dKB/s %d%%\n", progressReader.Name, KB, percent)
		} else {
			fmt.Printf("%s: %.2fMB/s %d%%\n", progressReader.Name, float64(KB)/1024, percent)
		}
		
		if progressReader.Done.Load() {
			ticker.Stop()
			return
		}
	}
}

func (progressReader *ProgressReader) Read(p []byte) (int, error) {
	n, err := progressReader.Reader.Read(p)
	progressReader.Progress.Add(int64(n))
	if !progressReader.Once {
		progressReader.Once = true
		go progressReader.NewLoop()
	}
	if err != nil {
		progressReader.Done.Store(true)
	}
	return n, err
}

func (progressReader *ProgressReader) Close() error {
	progressReader.Done.Store(true)
	return nil
}

This type will hold all the information we need: the name, the total size, current progress, whether the monitor goroutine has started and if the monitor goroutine should exit.

tip

We implemented the io.Closer interface because although http.NewRequest only needs an io.Reader, http.Transport will close the body after the request is done if the body is also an io.Closer. After the body is closed, the monitor goroutine will exit.

Restarting Slow Uploads

In practice, sometimes uploads become unreasonably slow. When this happens, I usually Ctrl + Z the upload process, then after some time resume it. The upload is interrupted, but because I have retry logic in case of error, the upload will be retried and usually the problem will go away.

note

If the slow transfer is due to network itself, obviously this won’t work.

Now that we know how to track the upload speed, we just need to set a slow threshold and if the speed is below the threshold for long enough, we cancel the request and retry it.

We could return an error as the body is being read. However, as golang will first buffer the body, then write it on the wire, it will have no use when the body is completely read.

Per golang documents, instead we should supply a context.Context in this case to control the entire lifetime of the request. This even works if the client has buffered all the body but failed to get a response in a timely manner.

It is very easy to adapt the above example to achieve the new requirements. It’s left as an exercise.

tip

The corresponding Contex.CancleFunc should be a member of the struct of the upload body. We also need to hold a reference to the Done channel of the context.Context to see if the monitor goroutine should exit.

Do not cancel the context when the body is closed - the request may not have been written yet.