The Linux fileset toolkit

Fast file operations.
Built for scale.

Copy files, remove directory trees, and manage permissions efficiently. For everyday work and millions of files on Linux.

Open source · MIT licensed · Linux x86_64 & AArch64

Familiar work. Larger scale.~/datasets
# copy a directory tree
$ rcp ./dataset ./dataset-copy \
    --progress --summary
# or send it to another host
$ rcp ./dataset host:/data/dataset \
    --auto-deploy-rcpd --progress
# remove an old scratch directory
$ rrm ./scratch/old-run \
    --progress --summary
# recursive removal, with progress
# and a summary when the work is done
# set a tree's group and permissions
$ rchm --group data \
    --mode "f:g+rw d:g+rwxs" \
    ./dataset --progress --summary
# separate rules for files and directories
For sysadmins, SREs, and systems engineers.From source trees and scratch space to shared datasets.
01 / THE ESSENTIALS

The work you do every day.

Three fundamental operations, with tools built to handle large directory trees.

rcp

Copy. Here or across hosts.

Copy files and directory trees locally or between Linux hosts. Preserve metadata, track progress, and tune concurrency when you need to.

Copy examples

rrm

Remove large directory trees.

Clear out scratch space, old datasets, and build output. Remove large filesets with progress reporting and a summary of the work.

Removal documentation

rchm

Put permissions in order.

Change ownership, groups, and permissions recursively. Apply different rules to files and directories in the same operation.

Permissions guide
02 / MEASURED PERFORMANCE

Less time waiting on a copy.

A million files is a useful place to start. In these shared-filesystem measurements, default rcp completes the copy in minutes.

Lustre
rcp · automatic settings8.7 min
dcp · 32 MPI ranks99.4 min
Weka
rcp · automatic settings6.1 min
dcp · 32 MPI ranks64.1 min

From the performance report. 1,024,000 files × 1 MB (1.024 TB), copied within each filesystem. Mean command wall time; lower is faster. These are two workloads from the report, not a general speedup claim. Filesystems use different hardware; the charts compare tools within each setting.

Useful for the smaller jobs, too.

On the tested Linux source tree, local rcp and cp both take about 2.5 seconds. The report also covers remote transfers, NFS, XFS, ZFS, and workloads where other settings or tools perform better.

03 / ROUND OUT YOUR TOOLKIT

Compare, link, and generate.

rcmp

Compare filesets and report differences between directory trees.

rlink

Build hard-linked filesets, with update options for datasets that change.

filegen

Generate repeatable sample filesets for testing and benchmarking.

04 / GET STARTED

Put the tools to work.

Install on Linux with Cargo or Nix, or get a Debian or RPM package from the releases.

Installation guide
Install the three core tools with Cargo
$ cargo install rcp-tools-rcp \
    rcp-tools-rrm rcp-tools-rchm
Make your first local copy
$ rcp ./dataset ./dataset-copy --progress
Read the destination-path and overwrite rules ↗