Adding Checksums to Golang Webdav Server Part I

Golang x/net has an implementation of WebDAV protocol, which can be used as a backend for Rclone. Rclone can be used to manage files on various storage systems, be it local or on cloud. Using WebDAV with Rclone, we can sync local files with files stored on remote server.

However, vanilla golang webdav server doesn’t support checksum, meaning rclone will only use file size to determine if file has changed. In some situations, rclone won’t be able to pick up local filesystem changes, resulting in data discrepancy.

Fortunately, rclone supports several quirks other webdav servers have, such as checksum and modified time support for OwnCloud and NextCloud.

Extending Golang Webdav

When instantiating a new webdav handler instance, go allows us to use a custom filesystem implementation. What we are interested in is DeadPropsHolder interface. This interface allows a webdav resource to return custom properties and modify them. As this resource is returned from webdav filesystem interface, we need to create a custom filesystem type as well.

How RClone Interacts with WebDAV

Inspecting RClone source code can reveal rclone attaches X-OC-Mtime header for PUT, COPY and MOVE requests, and OC-Checksum for PUT requests.

Storing Checksums

For simplicity’s sake, we store the checksums as an extended attribute of the file itself. Extended attributes are often used to provide additional functionality to a filesystem. There is a golang library to manipulate the extended attributes of a file.

Implementing Webdav Server with Checksum Support

Now that we understand how webdav server can be extended and how rclone interacts with owncloud webdav servers and decided how to store these checksums, we can then implement these methods.

Our custom types will embed corresponding webdav types anonymously so that we can focus on methods that needs to be modified.

davFS

davFS is the filesystem implementation, responsible for returning file resources.

OpenFile

Because our resources need to implement DeadPropsHolder interface, we’ll need to rewrite this method to return our custom file implementation.

func (fs davFS) OpenFile(ctx context.Context, name string, flag int, perm os.FileMode) (webdav.File, error) {
	file, err := fs.Dir.OpenFile(ctx, name, flag, perm)
	if err != nil {
		return nil, err
	}
	return &davFile{file, name}, nil
}

davFile

davFile is the actual resource the filesystem returns. It implements DeadPropsHolder interface. Because we delay the extraction of checksums, we have to save the resource path.

DeadProps

DeadProps returns the checksum of the file if found, the error is always nil.

func (f *davFile) DeadProps() (map[xml.Name]webdav.Property, error) {
	var (
		xmlName       xml.Name
		property      webdav.Property
		properties    = make(map[xml.Name]webdav.Property)
		checksum, err = readChecksum(f.name)
	)
	if err == nil {
		xmlName.Space = "http://owncloud.org/ns"
		xmlName.Local = "checksums"
		property.XMLName = xmlName
		property.InnerXML = append(property.InnerXML, "<checksum xmlns=\"http://owncloud.org/ns\">"...)
		property.InnerXML = append(property.InnerXML, checksum...)
		property.InnerXML = append(property.InnerXML, "</checksum>"...)
		properties[xmlName] = property
	}

	var stat fs.FileInfo
	stat, err = f.Stat()
	if err == nil {
		xmlName.Space = "DAV:"
		xmlName.Local = "lastmodified"
		property.XMLName = xmlName
		property.InnerXML = strconv.AppendInt(nil, stat.ModTime().Unix(), 10)
		properties[xmlName] = property
	}

	return properties, nil
}

readChecksum is a util function that extracts the checksum of the resource.

Patch

PATCH applies property modification to the resource. It is used under the hood during copy operations. Note we abuse golang xml to decode the inner xml to []byte:

func (f *davFile) Patch(proppatches []webdav.Proppatch) ([]webdav.Propstat, error) {
	var stat webdav.Propstat
	stat.Status = http.StatusOK
	for _, patch := range proppatches {
		for _, prop := range patch.Props {
			stat.Props = append(stat.Props, webdav.Property{XMLName: prop.XMLName})
			if prop.XMLName.Space == "http://owncloud.org/ns" && prop.XMLName.Local == "checksums" {
				var data []byte
				_ = xml.Unmarshal(prop.InnerXML, &data)
				if len(data) != 0 {
					updateChecksum(f.name, data)
				}
			} else if prop.XMLName.Space == "DAV:" && prop.XMLName.Local == "lastmodified" {
				modtimeUnix, err := strconv.ParseInt(string(prop.InnerXML), 10, 64)
				if err == nil {
					updateModtime(f.name, time.Unix(modtimeUnix, 0))
				}
			}
		}
	}
	return []webdav.Propstat{stat}, nil
}

updateChecksum is a util function that updates the checksum of a file. updateModtime is a util function that updates the modified time of the file.

To Be Continued

This part only covers how rclone interacts with owncloud webdav server and webdav optional interface implementation. The next part deals with how to extract the extra information from http headers and some caveats with the integration.