Application demands on storage have increased tremendously in recent years. Current applications in large data centers and high performance computing (HPC) environments demand high capacity (in the range of peta bytes), high IO throughput (greater than 200 GB/s) and large IOPs (Input/output per second). To cope with these demanding workloads, modern storage systems are now moving away from conventional SAN (Storage area network) and NAS (Network attached storage) models to highly parallel architectures.
DROP (Dynamic Ring Online Partitioning) is efficient and scalable distributed metadata management architecture to serve EB-scale file systems. It is a key-value store system in which primary key is a pathname. In order to keep excellent namespace locality, DROP exploits locality-preserving hashing to distribute metadata among metadata servers. When storage load changes dynamically, it introduces an efficient Histogram-based Dynamic Load Balancing (HDLB) strategy to quickly adjust the metadata distribution. After the adjustment, DROP ensures that the namespace locality maintained by metadata servers remains good.
Besides, DROP can balance the metadata storage load as good as static hash-based mapping. When the size of the metadata server cluster changes, DROP uses the HDLB strategy to move the minimal metadata to maintain the storage load balancing. It keeps excellent consistency of metadata replicas as well. Compared to other distributed metadata management techniques, DROP brings multiple advantages, such as balancing metadata storage load efficiently, high scalability and no bottlenecks with negligible additional overhead.
System architecture of DROP
Evaluation of DROP mechanism with various traces