Maybe I'm misunderstanding, but the deduplication of Borg happens at a repository level and after the client-side encryption. If several hosts are pushing archives containing the same files (or rather, file contents) into the same repository, they are deduplicated.
We rely on this in some places in fact. We have a couple of redundant hosts all serving the same file set. This deduplication allows us to push backups from all systems without coordination and only the first backup of the same file set requires storage beyond metadata.
> Another difference between the two programs is related to deduplication. Borg is designed with the assumption that each machine being backed up will use its own repository. Letting multiple machines backup to the same repository can impact performance, and simultaneous backups from different machines to the same respository are not supported.
So I should have said "Borg has limitations" instead of saying the support is absent for deduplication across machines.
When you say "without coordination", does it mean simultaneous backups from several machines to the same repo are possible?
> When you say "without coordination", does it mean simultaneous backups from several machines to the same repo are possible?
It is, with a bit of (other) coordination. Basically, only one process can actively write to a borg repository. Once something is writing to the borg repository, the repo is locked and nothing else can write to that repository.
However, you can easily stagger backups from several hosts into the same repository - node 1 writes at 09:00, node 2 writes at 10:00 and node 3 writes at 11:00. This works without problems, it just needs some monitoring for quickly growing backup sets in case your timing goes awry. You can also configure a wait timeout, how long a borg process will wait to lock the repository, but that will require some tinkering with SSH heartbeats to avoid connection timeouts, as borg won't talk over the wire while waiting.
As the documentation correctly states, this slows down the backup writing process a bit, because each node has to synchronize its local chunk cache at the start of a backup or a prune. This wouldn't be necessary if each node had their own dedicated backup repository. This can take 5ish minutes on our large repos (4TB+) and usually takes less than a minute for our smaller repos (<1TB). It's not really a big deal imo.
However - and we made that mistake earlier - something like `borg check` and other commands become really slow if you have some 15TB - 20TB+ borg repository and borg repos that large become really messy to manage. It works, but some things work at a glacial speed on a good day - while locking out all other backups. That's why we tend to group our borg repos by dataset and storage type ("All databases supporting app X backup into the app-database-X repository"). This way, all databases are in that repository even after failovers or switches to geo-standbys and it's easy to setup a restore procedure. Hoqever the overall size of the repository is somewhat limited and manageable.
Yes, all hosts accessing the borg repository need the encryption keys or the passphrase of the repository, since it relies on the homomorphic encryption of the repo.
And yeah, we tend to partition our borg repos along functionality and needs of restore. For example, database backups and file store backups are split into different repos, but several file store hosts in the same cluster all write to the same borg repo. After all, if something allowed to compromise one file store host, it will most likely allow compromise of all identically setup file store hosts in that cluster.
And on the other hand, in this way, we don't have to think about the backups if one of the file store hosts goes offline - the other two will just continue writing backups. This would be something that's really easy to forget and could bite very badly.
We rely on this in some places in fact. We have a couple of redundant hosts all serving the same file set. This deduplication allows us to push backups from all systems without coordination and only the first backup of the same file set requires storage beyond metadata.