A common observation is that backing up or synchronizing a large number of small files is significantly slower than transferring the same total amount of data as a few large files. For example, copying 100,000 files of 100 KB each (approximately 10 GB total) can take many times longer than copying a single 10 GB file. This article explains why this happens and what you can do about it.
Why Small Files Are Slow
Per-File Overhead
Every file that SyncBack processes requires a series of operations beyond simply reading and writing the file contents:
- Opening the file on the source
- Reading the file attributes, dates, and security information
- Creating or opening the file on the destination
- Writing the file contents
- Setting the file attributes, dates, and security information on the destination
- Closing the file on both source and destination
- Updating the log
- Updating the Fast Backup database (if enabled)
For a large file, this overhead is negligible compared to the time spent transferring the data. For a 1 KB file, the overhead can take longer than the actual data transfer.
Directory Scanning
Before any files are copied, SyncBack must scan the source and destination to build a list of files and compare them. Each directory must be opened and enumerated. For a folder structure with hundreds of thousands of directories, the scanning phase alone can take a significant amount of time, particularly on network storage where each directory listing requires a network round-trip.
Network Round-Trips
When working with remote storage (network shares, FTP, SFTP, or cloud), each file operation involves one or more network round-trips. A round-trip on a local network typically takes 0.5 to 2 milliseconds. This seems negligible for a single file, but for 100,000 files it adds up:
- 100,000 files multiplied by approximately 5 round-trips per file (open, read, write, set attributes, close) at 1 millisecond each equals approximately 500 seconds, or over 8 minutes, spent purely on network latency.
This is time spent waiting for responses, not transferring data. A single large file of the same total size would require only a handful of round-trips.
FTP and SFTP are particularly affected because each file operation is a separate command-response exchange. Cloud storage is even worse because each operation is an HTTPS request with TLS overhead.
Hard Drive Seek Time
Traditional hard drives (not SSDs) must physically move a read/write head to access each file. When processing many small files scattered across the disk, the drive spends more time seeking between files than actually reading data. This can reduce effective throughput from over 100 MB/s for sequential reads down to 1-5 MB/s for random small-file access.
SSDs do not have this problem because they have no moving parts and can access any location equally fast.
File System Metadata
The file system (typically NTFS on Windows) must look up each file in its Master File Table (MFT). For very large numbers of files, the MFT itself becomes large and lookups become slower. Directories with hundreds of thousands of entries are particularly slow to enumerate.
Antivirus Overhead
If real-time antivirus protection is enabled, each file that SyncBack opens is also scanned by the antivirus software. The per-file scanning overhead is relatively constant regardless of file size, so it disproportionately affects small files.
How Much Slower Can It Be?
As a rough illustration, consider transferring 10 GB of data over a 1 Gbps network connection to a NAS with a hard drive:
Scenario 1: One 10 GB file
- Transfer time: approximately 90 seconds
- Overhead: negligible (one open, one close)
- Total: approximately 90 seconds
Scenario 2: 100,000 files of 100 KB each (10 GB total)
- Transfer time: approximately 90 seconds (same total data)
- Per-file network overhead: approximately 500 seconds
- Per-file disk seek overhead: approximately 200 seconds (HDD)
- Total: approximately 13 minutes or more
The same amount of data takes roughly 8 to 10 times longer when split across many small files.
Recommendations
Use Fast Backup
Fast Backup allows SyncBack to skip files that have not changed since the last run. Instead of scanning and comparing every file, SyncBack uses a database of file modification dates and sizes from the previous run. This dramatically reduces the number of files that need to be processed on subsequent runs.
Use Compression
If your backup destination supports it, using a Zip or compressed backup consolidates many small files into a single archive. This reduces per-file overhead on the destination, especially for network and cloud storage. However, be aware that creating and updating the archive itself has overhead.
Reduce Directory Depth
Deeply nested folder structures with many directories increase scanning time. If you have control over the folder structure, flatter hierarchies are faster to scan.
Use an SSD on the Destination
If the destination is local or a NAS, using an SSD instead of a hard drive eliminates seek time, which is one of the largest contributors to small-file slowness.
Exclude Unnecessary Files
Use SyncBack's file and folder filters to exclude files that do not need to be backed up, such as temporary files, cache directories, and build output. Reducing the total file count directly reduces per-file overhead.
Consider the Network Protocol
If you are backing up to a NAS or server, using a Windows network share (SMB) is generally faster for many small files than FTP or SFTP, because SMB can pipeline multiple operations and supports compound requests. Cloud storage APIs are the slowest option for large numbers of small files due to per-request HTTPS overhead.