HDFS

At a glance

Capability	Support
rclone support tier	Tier 2: Stable
Checksums	None
Modification times	Kept, to 1 s
Case-insensitive names	No
Duplicate names allowed	No
Server-side copy	No
Server-side move and rename	Yes
Folder move and rename	Yes, in one step
Free space reporting	Yes
Empty folders	Yes
Change notifications	No
Uploads of unknown size	Yes
Public links	No

From rclone’s documentation

The rest of this page is adapted from rclone’s documentation for this backend (rclone v1.75.2, MIT licence). It is written for rclone’s command line: where it mentions a flag such as --hdfs-example, set the option example in the wizard’s options instead. Commands such as rclone copy aren’t part of CloudMount.

HDFS is a distributed file-system, part of the Apache Hadoop framework.

Setup notes

In CloudMount you add this drive from the Add drive screen, and the wizard asks for the options listed under Options. The notes below come from rclone’s setup guide.

Setting up your own HDFS instance for testing

You may start with a manual setup or use the docker image from the tests:

If you want to build the docker image

git clone https://github.com/rclone/rclone.git
cd rclone/fstest/testserver/images/test-hdfs
docker build --rm -t rclone/test-hdfs .

Or you can just use the latest one pushed

docker run --rm --name "rclone-hdfs" -p 127.0.0.1:9866:9866 -p 127.0.0.1:8020:8020 --hostname "rclone-hdfs" rclone/test-hdfs

NB it need few seconds to startup.

For this docker image the remote needs to be configured like this:

[remote]
type = hdfs
namenode = 127.0.0.1:8020
username = root

You can stop this image with docker kill rclone-hdfs (NB it does not use volumes, so all data uploaded will be lost.)

Modification times

Time accurate to 1 second is stored.

Checksum

No checksums are implemented.

Usage information

You can use the rclone about remote: command which will display filesystem size and current usage.

Restricted filename characters

In addition to the default restricted characters set the following characters are also replaced:

Character	Value	Replacement
:	0x3A	：

Invalid UTF-8 bytes will also be replaced.

Options

Standard options

The wizard asks for these when you add the drive.

namenode

Hadoop name nodes and ports.

E.g. “namenode-1:8020,namenode-2:8020,…” to connect to host namenodes at port 8020.

Type: CommaSepList

username

Hadoop user name.

Type: string

Choices:

• root: Connect to hdfs as root.

Advanced options

The wizard shows these under Advanced options. Most drives work without changing any of them.

service_principal_name

Kerberos service principal name for the namenode.

Enables KERBEROS authentication. Specifies the Service Principal Name (SERVICE/FQDN) for the namenode. E.g. "hdfs/namenode.hadoop.docker" for namenode running as service ‘hdfs’ with FQDN ‘namenode.hadoop.docker’.

Type: string

data_transfer_protection

Kerberos data transfer protection: authentication|integrity|privacy.

Specifies whether or not authentication, data signature integrity checks, and wire encryption are required when communicating with the datanodes. Possible values are ‘authentication’, ‘integrity’ and ‘privacy’. Used only with KERBEROS enabled.

Type: string

Choices:

• privacy: Ensure authentication, integrity and encryption enabled.

encoding

The encoding for the backend.

See the encoding section in the overview for more info.

Type: Encoding · Default: Slash,Colon,Del,Ctl,InvalidUtf8,Dot

description

Description of the remote.

Type: string

Limitations

• Erasure coding not supported, see issue #8808
• No server-side Move or DirMove.
• Checksums not implemented.