note
This article was last updated on December 10, 2023, 9 months ago. The content may be out of date.
I wrote a simple cli tool to upload files to my Cloudreve instance. It’s easy to figure out which api to call when uploading files. However, it will be helpful to know how the upload is going.
Visualizing Speed
When creating an upload request, golang allows us to use a io.Reader
as the request body. The body will be responsible for displaying upload progress.
info
Although Cloudreve and its storage providers require Content-Length
header to be set, thus requiring the total length to be known beforehand, this method is applicable even if the total is unknown. We lose out on the ability to track the percentage of the upload in this case though 🤷.
type ProgressReader struct {
Reader io.Reader
Progress atomic.Int64
Size int64
Name string
Once bool
Done atomic.Bool
}
func (progressReader *ProgressReader) NewLoop() {
ticker := time.NewTicker(time.Second)
var op int64
for range ticker.C {
p := progressReader.Progress.Load()
KB := (p - op) / 1024
var percent int64
if progressReader.Size != 0 {
percent = p * 100 / progressReader.Size
} else {
percent = 100
}
if KB < 1024 {
fmt.Printf("%s: %dKB/s %d%%\n", progressReader.Name, KB, percent)
} else {
fmt.Printf("%s: %.2fMB/s %d%%\n", progressReader.Name, float64(KB)/1024, percent)
}
if progressReader.Done.Load() {
ticker.Stop()
return
}
}
}
func (progressReader *ProgressReader) Read(p []byte) (int, error) {
n, err := progressReader.Reader.Read(p)
progressReader.Progress.Add(int64(n))
if !progressReader.Once {
progressReader.Once = true
go progressReader.NewLoop()
}
if err != nil {
progressReader.Done.Store(true)
}
return n, err
}
func (progressReader *ProgressReader) Close() error {
progressReader.Done.Store(true)
return nil
}
This type will hold all the information we need: the name, the total size, current progress, whether the monitor goroutine has started and if the monitor goroutine should exit.
tip
We implemented the io.Closer
interface because although http.NewRequest
only needs an io.Reader
, http.Transport
will close the body after the request is done if the body is also an io.Closer
. After the body is closed, the monitor goroutine will exit.
Restarting Slow Uploads
In practice, sometimes uploads become unreasonably slow. When this happens, I usually Ctrl + Z
the upload process, then after some time resume it. The upload is interrupted, but because I have retry logic in case of error, the upload will be retried and usually the problem will go away.
note
If the slow transfer is due to network itself, obviously this won’t work.
Now that we know how to track the upload speed, we just need to set a slow threshold and if the speed is below the threshold for long enough, we cancel the request and retry it.
We could return an error as the body is being read. However, as golang will first buffer the body, then write it on the wire, it will have no use when the body is completely read.
Per golang documents, instead we should supply a context.Context
in this case to control the entire lifetime of the request. This even works if the client has buffered all the body but failed to get a response in a timely manner.
It is very easy to adapt the above example to achieve the new requirements. It’s left as an exercise.
tip
The corresponding Contex.CancleFunc
should be a member of the struct of the upload body. We also need to hold a reference to the Done
channel of the context.Context
to see if the monitor goroutine should exit.
Do not cancel the context when the body is closed - the request may not have been written yet.