In this blog we will discuss the facts of availability using math and demystify the myth behinds ScaleIO’s high availability.
For data loss or data unavailability to occur in a system with two replicas of data (such as ScaleIO) there must be two concurrent failures or a second failure must occur before the system recovers from a first failure. Therefore one of the following four scenarios must occur:
1. Two drive failures in a storage pool OR
2. Two nodes failures in a storage pool OR
3. A node failed followed by a drive failure OR
4. A drive failed followed by a node failure
Let us choose two popular ScaleIO configurations and derive the availability of each.
20 x ScaleIO servers deployed on Dell EMC’s PowerEdge Servers R740xd with 24 SSD drives each, 1.92TB SSD drive size using 4 x 10GbE Network. In this configuration we will assume that the rebuild time is network bound.
20 x ScaleIO servers deployed on Dell EMC’s PowerEdge Servers R640 with 10 SSD drives each, 1.92TB SSD drives using 2 x 25GbE Network. In this configuration we will assume that the rebuild time is SSD bound.
Note: ScaleIO best practices recommend a maximum of 300 drives in a storage pool, therefore for the first configuration we will configure two storage pools with 240 drives in each pool.
To calculate the availability of a ScaleIO system we will leverage a couple of well know academic publications:
We will adjust the formulas in the paper to the ScaleIO architecture and model the different failures.
Two Drive Failures
We will use the following formula to calculate the MTBF of ScaleIO system for a two drive failure scenario:
Where:
◈ N = Number of drives in a system
◈ G = Number of drives in a storage pool
◈ M = Number of drives per server
◈ K = 8,760 hours
( 1 Year)
◈ = MTBF of a single drive
◈ = Mean Time to Repair – repair/rebuild time of a failed drive
Note: This formula assumes that two drives that fail in the same ScaleIO SDS (server) will not cause DU/DL as the ScaleIO architecture guarantees that replicas of the same data will NEVER reside on the same physical node.
Let’s assume two scenarios – in the first scenario the rebuild process is constrained by network bandwidth – in the second scenario the rebuild process is constrained by drive performance bandwidth.
Network Bound
In this case we assume that the rebuild time/performance is limited by the availability of network bandwidth. This will be the case if you deploy a dense configuration such as the DELL 740xd servers with a large number of SSDs in a single server. In this case, the MTTR function is:
Where:
◈ S – Number of servers in a ScaleIO cluster
◈ Network Speed – Bandwidth in GB/s available for rebuild traffic (excluding application traffic)
◈ Conservative_Factor = factor additional time to complete the rebuild (to be conservative).
Plugging in the relevant values in the formula above, we get a MTTR of ~1.5 minutes for the 20 x R740, 24 SSDS @ 1.92TB w/ 4 X 10GbE network connections configuration (two storage pools w/ 240 drives per pool). The 20 x R640, 10SSDs @ 1.92TB w/ 2 X 25GbE network connections config provides MTTR of ~2 minutes. These MTTR values reflect the superiority of ScaleIO’s declustered RAID architecture that result in a very fast rebuild time. In a later post we will show how those MTTR values are critical and how they impact system availability and operational efficiency.
SSD Drive Bound
In this case, the rebuild time/performance is bound by the number of SSD drives and the rebuild time is a function of the number of drives available in the system. This will be the case if you deploy less dense configurations such as the 1U Dell EMC PowerEdge R640 servers. In this case, the MTTR function is:
Where:
◈ G – Number of drives in a storage pool
◈ Drive_Speed – Drive speed available for rebuild
◈ Conservative_Factor = factor additional time to complete the rebuild (to be conservative).
System availability is calculated by dividing the time that the system is available and running, by the total time the system was running added to the restore time. For availability we will use the following formula:
Where:
◈ RTO – Recovery Time Objective or the amount of time it takes to recover a system after a data loss event (For example: if two drives fail in a single pool), where data needs to be recovered from a backup system. We will be highly conservative and will consider Data Unavailability (DU) scenarios as bad as Data Loss (DL) scenarios therefore we will use RTO in the availability formula.
Note: the only purpose of RTO is to translate MTBF to availability.
Node and Device Failure
Next, let’s discuss the system’s MTBF when a node fails and followed by a drive failure, for this scenario we will be using the followed model:
Where:
◈ M = Number of drives per node
◈ G = Number of drives in the pool
◈ S = Number of servers in the system
◈ K = Number of hours in 1 year i.e. 8,760 hours
◈ MTBFdrive = MTBF of a single drive
◈ MTBFserver = MTBF of a single node
◈ MTTRserver = repair/rebuild time of failed server
In a similar way, one can develop the formulas for other failure sequences such as a drive failure after a node failure and a second node failure after a first node failure.
Network Bound Rebuild Process
In this case we assume that rebuild time/performance is constrained by network bandwidth. We will make similar assumptions as for drive failure. In this case, the MTTR function is:
Where:
◈ M – Number of drives per server
◈ S – Number of servers in a ScaleIO cluster
◈ Network Speed – Bandwidth in GB/s available for rebuild traffic (excluding application traffic)
◈ Conservative_Factor = factor additional time to complete the rebuild to be conservative
Plugging the relevant values in the formula above, we get a MTTR of ~30 minutes for the 20 x R740, 24 SSDS @ 1.92TB w/ 4 X 10GbE network connections configuration (two storage pools w/ 240 drives per pool). The 20 x R640, 10SSDs @ 1.92TB w/ 2 x 25GbE Network config provides MTRR of ~20 minutes. During system recovery ScaleIO rebuilt about 48TB of data for the first configuration and about 21TB for the second configuration.
SSD Drive Bound
In this case we assume that the Rebuild time/performance is SSD drive bound and the rebuild time is a function of the number of drives available in the system. Using the same assumptions as for drive failures, the MTTR function is:
◈ G – Number of drives in a storage pool
◈ M – Number of drives per server
◈ Drive_Speed – Drive speed available for rebuild
◈ Conservative_Factor = factor additional time to complete the rebuild to be conservative
Based on the provided formulas let’s calculate the availability of ScaleIO system based on the two different configurations:
20 x R740, 24 SSDS @ 1.92TB w/ 4 X 10GbE Network
(Deploying 2 storage pools w/ 240 drives per pool)
| Reliability (MTBF) | Availability | |
| Drive After Drive | 43,986 [Years] | 0.999999955 |
| Drive After Node | 6,404 [Years] | 0.999999691 |
| Drive After Drive | 138,325 [Years] | 0.999999985 |
| Drive After Node | 38,424 [Years] | 0.999999897 |
| Overall System | 4,714 [Years] | 0.99999952 or 6-9’s |
20 x R640, 10SSDs @ 1.92TB w/ 2 x 25GbE:
| Reliability (MTBF) | Availability | |
| Drive After Drive | 105,655 [Years] | 0.999999983 |
| Drive After Node | 27,665 [Years] | 0.999999637 |
| Node After Drive | 276,650 [Years] | 0.999999993 |
| Drive After Node | 69,163 [Years] | 0.999999975 |
| Overall System | 15,702 [Years] | 0.99999989 or 6-9’s |
Since these calculations are complex, ScaleIO provides its customers with FREE online tools to build HW configurations and obtain availability numbers that includes all possible failure scenarios. We advise customers to use this tool, rather than crunch complex mathematics, to build system configurations based on desired system availability targets.
Posts
0 comments:
Post a Comment