Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Does Dropbox do block or file-level deduping?


Block, it deduplicates based on 4MB blocks.

Source: https://news.ycombinator.com/item?id=2478595


I wonder if they use a rolling checksum too, to avoid duplicating a complete file if only a view bytes shifted (for example adding a line of text in the beginning of a file)

The backup tool bup (https://github.com/bup/bup) does this.


They almost certainly do not, mostly because of how slow doing so is.


It probably wouldn't hit the most important cases either, dedup is typically most powerful & valuable on large media files, software packages, disk ISO's, and the like which do not frequently have arbitrary text inserted at the start of the file!




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: