SmartDedup: Optimizing deduplication for resource-constrained devices

Qirui Yang, Runyu Jin, Ming Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

18 Scopus citations


Storage on smart devices such as smartphones and the Internet of Things has limited performance, capacity, and endurance. Deduplication has the potential to address these limitations by eliminating redundant I/Os and data, but it must be considered under the various resource constraints of the devices. This paper presents SmartDedup, a deduplication solution optimized for resource-constrained devices. It proposes a novel architecture that supports symbiotic in-line and out-of-line deduplication to take advantage of their complementary strengths and allow them to be adapted according to a device’s current resource availability. It also cohesively combines in-memory and on-disk fingerprint stores to minimize the memory overhead while achieving a good level of deduplication. SmartDedup is prototyped on EXT4 and F2FS and evaluated using benchmarks, workloads generated from real-world device images, and traces collected from real-world devices. The results show that SmartDedup substantially improves I/O performance (e.g., increases write and read throughput by 31.1% and 32%, respectively for an FIO experiment with 25% duplication ratio), reduces flash writes (e.g., by 70.9% in a trace replay experiment with 75.8% duplication ratio), and saves space usage (e.g., by 45% in a DEDISbench experiment with 46.1% duplication ratio) with low memory, storage, and battery overhead, compared to both native file systems and related deduplication solutions.

Original languageEnglish (US)
Title of host publicationProceedings of the 2019 USENIX Annual Technical Conference, USENIX ATC 2019
PublisherUSENIX Association
Number of pages14
ISBN (Electronic)9781939133038
StatePublished - 2019
Event2019 USENIX Annual Technical Conference, USENIX ATC 2019 - Renton, United States
Duration: Jul 10 2019Jul 12 2019

Publication series

NameProceedings of the 2019 USENIX Annual Technical Conference, USENIX ATC 2019


Conference2019 USENIX Annual Technical Conference, USENIX ATC 2019
Country/TerritoryUnited States

ASJC Scopus subject areas

  • General Computer Science


Dive into the research topics of 'SmartDedup: Optimizing deduplication for resource-constrained devices'. Together they form a unique fingerprint.

Cite this