Hasher

In CloudMount

• Needs Pro. See Wrapping and combining drives.

At a glance

rclone support tier: Tier 4: Experimental

This is a wrapping drive: it stores nothing itself and works on top of other drives, so checksums, modification times, server-side operations and free space mostly follow the drives it wraps. The sections below say where it differs.

From rclone’s documentation

The rest of this page is adapted from rclone’s documentation for this backend (rclone v1.75.2, MIT licence). It is written for rclone’s command line: where it mentions a flag such as --hasher-example, set the option example in the wizard’s options instead. Commands such as rclone copy aren’t part of CloudMount.

Hasher is a special overlay backend to create remotes which handle checksums for other remotes. It’s main functions include:

• Emulate hash types unimplemented by backends
• Cache checksums to help with slow hashing of large local or (S)FTP files
• Warm up checksum cache from external SUM files

Getting started

To use Hasher, first set up the underlying remote following the configuration instructions for that remote. You can also use a local pathname instead of a remote. Check that your base remote is working.

Let’s call the base remote myRemote:path here. Note that anything inside myRemote:path will be handled by hasher and anything outside won’t. This means that if you are using a bucket based remote (S3, B2, Swift) then you should put the bucket in the remote s3:bucket.

Now proceed to interactive or manual configuration.

Interactive configuration

Run rclone config:

No remotes found, make a new one\?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> Hasher1
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Handle checksums for other remotes
   \ "hasher"
[snip]
Storage> hasher
Remote to cache checksums for, like myremote:mypath.
Enter a string value. Press Enter for the default ("").
remote> myRemote:path
Comma separated list of supported checksum types.
Enter a string value. Press Enter for the default ("md5,sha1").
hashsums> md5
Maximum time to keep checksums in cache. 0 = no cache, off = cache forever.
max_age> off
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
--------------------
[Hasher1]
type = hasher
remote = myRemote:path
hashsums = md5
max_age = off
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

Manual configuration

Run rclone config path to see the path of current active config file, usually YOURHOME/.config/rclone/rclone.conf. Open it in your favorite text editor, find section for the base remote and create new section for hasher like in the following examples:

[Hasher1]
type = hasher
remote = myRemote:path
hashes = md5
max_age = off

[Hasher2]
type = hasher
remote = /local/path
hashes = dropbox,sha1
max_age = 24h

Hasher takes basically the following parameters:

• remote is required
• hashes is a comma separated list of supported checksums (by default md5,sha1)
• max_age - maximum time to keep a checksum value in the cache 0 will disable caching completely off will cache “forever” (that is until the files get changed)

Make sure the remote has : (colon) in. If you specify the remote without a colon then rclone will use a local directory of that name. So if you use a remote of /local/path then rclone will handle hashes for that directory. If you use remote = name literally then rclone will put files in a directory called name located under current directory.

Usage

Basic operations

Now you can use it as Hasher2:subdir/file instead of base remote. Hasher will transparently update cache with new checksums when a file is fully read or overwritten, like:

rclone copy External:path/file Hasher:dest/path
rclone cat Hasher:path/to/file > /dev/null

The way to refresh all cached checksums (even unsupported by the base backend) for a subtree is to re-download all files in the subtree. For example, use hashsum --download using any supported hashsum on the command line (we just care to re-read):

rclone hashsum MD5 --download Hasher:path/to/subtree > /dev/null
rclone backend dump Hasher:path/to/subtree

You can print or drop hashsum cache using custom backend commands:

rclone backend dump Hasher:dir/subdir
rclone backend drop Hasher:

Pre-Seed from a SUM File

Hasher supports two backend commands: generic SUM file import and faster but less consistent stickyimport.

rclone backend import Hasher:dir/subdir SHA1 /path/to/SHA1SUM [--checkers 4]

Instead of SHA1 it can be any hash supported by the remote. The last argument can point to either a local or an other-remote:path text file in SUM format. The command will parse the SUM file, then walk down the path given by the first argument, snapshot current fingerprints and fill in the cache entries correspondingly.

• Paths in the SUM file are treated as relative to hasher:dir/subdir.
• The command will not check that supplied values are correct. You must know what you are doing.
• This is a one-time action. The SUM file will not get “attached” to the remote. Cache entries can still be overwritten later, should the object’s fingerprint change.
• The tree walk can take long depending on the tree size. You can increase --checkers to make it faster. Or use stickyimport if you don’t care about fingerprints and consistency.

rclone backend stickyimport hasher:path/to/data sha1 remote:/path/to/sum.sha1

stickyimport is similar to import but works much faster because it does not need to stat existing files and skips initial tree walk. Instead of binding cache entries to file fingerprints it creates sticky entries bound to the file name alone ignoring size, modification time etc. Such hash entries can be replaced only by purge, delete, backend drop or by full re-read/re-write of the files.

Configuration reference

Options

Standard options

The wizard asks for these when you add the drive.

remote

Remote to cache checksums for (e.g. myRemote:path).

Type: string · Required

hashes

Comma separated list of supported checksum types.

Type: CommaSepList · Default: md5,sha1

max_age

Maximum time to keep checksums in cache (0 = no cache, off = cache forever).

Type: Duration · Default: off

Advanced options

The wizard shows these under Advanced options. Most drives work without changing any of them.

auto_size

Auto-update checksum for files smaller than this size (disabled by default).

Type: SizeSuffix · Default: 0

description

Description of the remote.

Type: string

Implementation details (advanced)

This section explains how various rclone operations work on a hasher remote.

Disclaimer. This section describes current implementation which can change in future rclone versions!.

Hashsum command

The rclone hashsum (or md5sum or sha1sum) command will:

1. if requested hash is supported by lower level, just pass it.
2. if object size is below auto_size then download object and calculate requested hashes on the fly.
3. if unsupported and the size is big enough, build object fingerprint (including size, modtime if supported, first-found other hash if any).
4. if the strict match is found in cache for the requested remote, return the stored hash.
5. if remote found but fingerprint mismatched, then purge the entry and proceed to step 6.
6. if remote not found or had no requested hash type or after step 5: download object, calculate all supported hashes on the fly and store in cache; return requested hash.

Other operations

• any time a hash is requested, follow the logic from 1-4 from hashsum above
• whenever a file is uploaded or downloaded in full, capture the stream to calculate all supported hashes on the fly and update database
• server-side move will update keys of existing cache entries
• deletefile will remove a single cache entry
• purge will remove all cache entries under the purged path

Note that setting max_age = 0 will disable checksum caching completely.

If you set max_age = off, checksums in cache will never age, unless you fully rewrite or delete the file.

Cache storage

Cached checksums are stored as bolt database files under rclone cache directory, usually ~/.cache/rclone/kv/. Databases are maintained one per base backend, named like BaseRemote~hasher.bolt. Checksums for multiple alias-es into a single base backend will be stored in the single database. All local paths are treated as aliases into the local backend (unless encrypted or chunked) and stored in ~/.cache/rclone/kv/local~hasher.bolt. Databases can be shared between multiple rclone processes.