Splunking Responsible part 2: How to get the right storage capacity (2023)

Without going into too much detail, the benefit of warm -> warm -> cold -> frozen is that storage can be tuned to best suit the use case required by the environment and storage costs can be managed.

preservation

Warm Storage is where the hot and tepid barrels live. It is also the only storage where new/incoming data is written. This storage type should be the fastest storage type available for your Splunk system: Splunk requires a minimum of 800 IOPS for this storage. It makes sense to go for NVMe or SSD drives for this, if they're available. If you're using spinning drives, you'll generally need RAID 10 to achieve acceptable performance (in my experience, you need an 8-drive array to achieve those numbers on all but the fastest 15k SAS drives). SAN storage is perfectly acceptable as long as it can handle the amount of IOPS Splunk needs (test your storage first, don't just rely on your SAN provider). Shared storage via NFS should never be used with warm Splunk.

That being said, assuming we use a flash backup system as our hot storage, we may not be able to store everything on there due to the cost of those drives. This is where understanding our Splunk use case comes in: for example, if our primary use case for Splunk is a Security Operations Center (SOC), we may find that the vast majority of queries fail a week in contains data. This means that you can consider using very fast storage for these amounts of data, and understand that longer queries use slower storage and will not complete quickly.

cold storage

Cold storage is the second type of storage defined in Splunk and the last type that is actively searchable. Cold storage is generally used for data that can be searched without additional steps (such as in the PCI "out of the box" requirement), but the degradation in search performance is acceptable. This storage type is often used as a trade-off, especially when using a high cost/low capacity storage type, such as an NVMe SSD solution backend, for Splunk and keeping all searchable data on this type is a pain. inadequate actual storage. .

In some cases, you can simply assign a large storage pool to Splunk. In this case, there is no difference in performance between hot and cold storage. Splunk still rolls too cold buckets under the hood, so you only want to symbolize cold mount points for warm storage.

I haven't been able to find official recommendations on performance requirements for cold storage because unless you're using flash-based storage, it's generally more difficult to get the required storage performance from hot storage. At the very least, it should be noted that this storage must be good enough to perform the bucket rotation activity (copying index data from hot to cold) and all queries that use this data.

Since cold storage is lower-level storage, it's tempting to use network-based storage like NFS for this (NFS *doesn't* support hot storage, but cold storage does). Note that this exposes your environment to stability issues due to NFS failures: I have seen full indexers fail due to NFS cold storage access issues (leading to cascading Splunk infrastructure failures).

frozen storage

The third type of storage is significantly less convenient than hot or cold storage and can be used as a cost control solution when longer retention periods are required for regulatory or compliance reasons.

By default, Splunk does not use frozen storage: the freeze behavior deletes data once the configured cold retention period has expired. Splunk administrators can override this behavior to indicate where data is stored when scrolling to freeze.

The advantage of frozen data is that it takes up much less space than other data indexed by Splunk, typically around 15% of the size of the original data. If you thought that number sounded familiar, you were right: In the first calculation we did to determine how much space the raw data would take up when the data was indexed, it was the same as 15%. Roll to Freeze didn't magically give us more space: we removed the searchable metadata (35% of the original data size), making our data unsearchable in Splunk, and archived the raw data compressed.

For frozen data to be searchable in Splunk, the data must be rolled back. This means that the data is moved to a location in the indexer where it can be read, and Splunk rebuilds the necessary metadata to make the data searchable. Depending on the data, this can be a very time-consuming (and manual) process.

Once Splunk freezes the data, it is no longer tracked. A process outside of Splunk should be used to clean up frozen data (for example, delete it after the retention period has expired).

Therefore, frozen data is best suited as a file that is not expected to be accessed during normal business operations, but must be retained for legal or compliance reasons. Regular access to frozen data in Splunk is not a productive use of an administrator's time.

Acceleration of data models

An often overlooked component of storage compute is the space required to store data model acceleration data, which is typically consumed by Splunk Enterprise Security Suite (ES). The official calculation of this data is 3.4 times the daily usage to calculate the storage capacity of the accelerated data model data for one year. This means that one year of data model acceleration with a 10 GB/day license takes approximately 34 GB of disk space. This is not a huge consideration for small deployments, but it increases as the environment scales.

example

Now that we've covered the storage classes, let's look at some examples, starting with a small environment of 10 GB/day and then expanding to the example environment described in part 1 of the Splunking Responsibly blog (500 GB/day).

Storage calculation: single instance, 10 GB/day

Let's start with a simple example:

• Splunk license capacity: 10 GB/day
• Single server, no replication

With this capacity, we expect to consume 5 GB of actual disk space per day:

• 10 GB/day * 0.15 (raw data file) = 1.5 GB/day of disk usage
• 10 GB/day * 0.35 (metadata lookup) = 3.5 GB/day of disk usage
• 1.5 GB + 3.5 GB = 5 GB/day of disk usage

Now that we know how much space we need per day, we can use the reserved numbers to extrapolate our storage needs. Assuming we keep everything in either hot or cold storage, here's a simple calculation:

• Search*day by day

In our 10 GB/day example:

90 day retention:

• 5GB daily use * 90 days = 450GB disk - 1 year reserve:
• 5 GB of daily usage * 365 days = 1825 GB of disk

Acceleration data model (assuming 1 year):

• 10 GB * 3.4 = 34 GB

1 year retention + DMA:

• 1825 GB + 34 GB = 1859 GB

Storage Suggestions:

In this example, I recommend that the customer allocate 2TB of space for Splunk data.

If we want to reduce storage requirements by using frozen data, but need to keep 90 days of "scan-ready" data and 1 year of usable data (according to PCI DSS 10.7), our calculation would look like this:

Cold/hot storage:

• 5 GB daily usage * 90 days = 450 GB

Freezer Storage:

• 1.5 GB of daily usage * 275 days = 412.5 GB

total:

• 450 GB (hot/cold) + 412.5 GB (frozen) + 34 GB (DMA) = 896.5 GB

Storage Suggestions:

• In this example, I recommend that the customer allocate 1TB of space for Splunk data.

As you can see, using the freeze to archive 9-month data significantly reduces storage requirements at the cost of making the data readily available.

Storage compute: single instance, 500 GB/day + cluster

Now look at a more complex example. Here we are referring to the 500 GB/day implementation from my previous article. The environment consists of 8 indexers. For the sake of this example, we assume perfect data distribution across all indexers, with a 2:2 search and replication factor per site.

Let's start by calculating the basic storage requirements:

• 500 GB/day * 0.15 (raw data file) = 75 GB/day of disk usage
• 500 GB/day * 0.35 (metadata lookup) = 175 GB/day of disk usage
• 75 GB + 175 GB = 250 GB/day of disk usage (before copy)
• 75 GB * 2 (replication factor) + 175 GB * 2 (seek factor) = 500 GB/day
• 500 GB/day/8 indexers (assuming unified distribution) = 62.5 GB/day/indexer

At this point, we move on to the calculations based on the desired reserve:

90 day retention:

• 500 GB of daily usage * 90 days = 45 TB of disk
• 45 TB / 8 indexers = 5625 TB disk per indexer

1 year retention:

• 500 GB of daily usage * 365 days = 182.5 TB of disk
• 182.5TB/8 indexers = ~23TB disk per indexer

Acceleration data model (assuming 1 year):

• 500GB * 3.4 = 1700GB

1 year retention + DMA:

• 182,5 TB + 1,7 TB = 184,2 TB
• 184.2 TB/8 indexers = about 23 TB of disk per indexer (a bit more)

Storage Suggestions:

• In this example, I recommend that customers allocate more than 23TB of space for Splunk data on each indexer.

This gets more complicated if you use calculations of 90 hot/cold days + 275 freezing days, but since we've done everything we've done so far, why not?

Cold/hot storage:

• 500 GB of daily usage * 90 days = 45 TB of disk
• 45 TB / 8 indexers = 5625 TB disk per indexer

Freezer Storage:

• 75 GB of daily usage * 275 days = 20,625 TB of disk
• 20 625 TB / 8 indexers = 2578 TB disk per indexer

total:

• Full environment: - 45TB (hot/cold) + 20,625TB (frozen) + 1.7TB (DMA) = 67,325TB
• Per indexer: - 5.625 TB (hot/cold) + 2.578 TB (frozen) + 0.2125 TB (DMA) = ~8.42 TB per indexer

Storage Suggestions:

• In this example, I recommend that customers allocate approximately 9TB of space for Splunk data per indexer.

But math is hard!

It's important to understand how storage is calculated and to be able to do it manually if needed (I've used many whiteboards to do this example with clients, and if I could do it manually it would really help them "get it"). With that said, once you're comfortable with the process, there are tools out there that can make the process much easier.

The best tool I have found so far ishttps://splunk-sizing.appspot.com/This doesn't cover all possible combinations (such as some daily consumption values ​​or calculations accelerated by the data model), but it will give you a quick estimate of how much space your client implementation will require.

That means, just like in school, being able to show your work and defend your calculations if a client has questions.