Hadoop的存储策略

Hadoop的存储策略 策略 ID策略名称块分布creationFallbacksreplicationFallbacks15Lazy_PersistRAM_DISK: 1, DISK: n-1DISKDISK12All_SSDSSD: nDISKDISK10One_SSDSSD: 1, DISK: n-1SSD, DISKSSD, DISK7Hot (default)DISK: n< none >ARCHIVE5WarmDISK: 1, ARCHIVE: n-1ARCHIVE, DISKARCHIVE, DISK2ColdARCHIVE:n< none >< none >

策略ID数值越大其存取速度越快

1、查看机器支持的异构存储策略

#查看机器支持的异构存储策略hdfs storagepolicies -listPolicies#----------------------------------------------------------------------------------------Block Storage Policies:BlockStoragePolicy{PROVIDED:1, storageTypes=[PROVIDED, DISK], creationFallbacks=[PROVIDED, DISK], replicationFallbacks=[PROVIDED, DISK]}BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]}BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]}BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]}#----------------------------------------------------------------------------------------

2、COLD

COLD存储策略主要用于存储冷数据，存储类型是ARCHIVE

BlockStoragePolicy{ COLD:2, storageTypes=[ARCHIVE],creationFallbacks=[],replicationFallbacks=[]}

3、Warm

Warm存储策略主要用于存储半热半冷数据，存储类型是DISK和ARCHIVE

BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]}

4、HOT

HOT存储策略主要用于存储热数据，存储类型是DISK

BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}

5、ONE_SSD

ONE_SSD的存储策略就是将一个主数据放在SSD介质上，其他备份文件放在DISK上。这是HDFS的默认数据存储策略

BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]}

6、ALL_SSD

这个数据的存储策略就是，将数据全部存储在SSD介质上，包括备份数据的数据，当SSD空间不足时才采取将数据放在DISK上

BlockStoragePolicyALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}

7、LAZY_PERSIST

LAZY_PERSIST的存储策略就是将新建的数据块放在RAM_DISK上，而备份的数据放在DISK上。RAM_DISK为随机读写磁盘，其是内存的主要构成单元，因此LAZY_PERSIST存储策略的数据其读写速度最快。

BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]}