Machine Learning-Based Prediction of Storage-Feasible Content-Defined Chunking Configurations for Data Deduplication
✍️ Authors
Hashem Bedr JehlolCorresponding
📖 Abstract
Content-defined chunking (CDC) preserves reusable regions after file modifications, but the storage benefit and metadata cost depend strongly on the selected chunking parameters. This study evaluates whether a machine-learning model can predict, before a later file version is processed, whether a deterministic CDC configuration will remain close to a storage-oriented reference. Five Gear-based configurations (C0-C4) were evaluated, with C0 used as the reference. A candidate was labeled storage-feasible when its loss in Version-B storage saving did not exceed 1.0 percentage point relative to C0. The model used 33 inputs available from Version A only. Training used 632 changed pairs from Linux 3.16, Linux 5.5, and backup version families. Extra Trees was selected by grouped five-fold out-of-fold evaluation and then frozen. On unseen Linux 5.7, the model achieved ROC-AUC 0.9419, PR-AUC 0.9121, precision 0.9924, and MCC 0.8534. On the independent 3DRDF final test, ROC-AUC was 0.8005, PR-AUC 0.8710, precision 0.9356, and MCC 0.6060. The fixed configurations exposed a clear storage-metadata trade-off: on 3DRDF, C0 saved 63.3353% of Version-B bytes, whereas C4 reduced the A+B chunk count by 76.8461% but saved 41.7543%.