Long Term Retention Tips and Tricks

Long Term retention (Hindsight) is a useful feature to allow for a cost-effective way to store data long-term. You will effectively pay S3 prices to store it and will be charged only when you query it. Long-term retention begins when your short-term retention period, which allows for unlimited queries, ends. 

How do I enable Hindsight?
Note that Hindsight is only available to customers with an annual agreement. You will need to set this up ahead of time, so please reach out to support@scalyr.com or reach out to your Customer Success or Account representative.

How long can I store data?

There is effectively no limit, so whether you want to save for 6 months or 6 years, DataSet is capable of storing your data. 

The LTR Retention period will need to be configured ahead of time. 

How much does it cost?

$0.022 per compressed GB / month stored

+

$0.50 per TB scanned.  

Query Cost Examples:

Example 1

plan - 30GB/Day plan ingesting ~30GB/day 

query - serverHost = *  

query duration - 365 days

data scanned -  10950GB

search cost - $5.48 

Example 2

plan - 30GB/Day plan ingesting ~30GB/day 

query - serverHost = "server1"

query duration - 365 days

data scanned -  10950GB

search cost - $5.48 

*storage cost not included in calculations

How is long-term retention billed?

• Main search page -  charged once per search 
• Graph page - charged once per search
• Breakdown graphs - charged twice per search
• Distribution view - charged once per search 
• PowerQuery - charged once 
• PowerQuery Join or Union - charged n per search (where n = # of queries)
• Alerts: free
• Dashboards: 
  • Multiplot - charged once the first render after that free
  • breakdown graph - charged once
  • powerquery - charged once (unless join or union)
  • report charged n (where n = # of columns)

Viewing graphs and PowerQueries causes time series to be created. This is why these are only charged once. There are no additional charges after the time series is created. When using the filter bar at the top of the dashboard, time series are not created, so each of these may result in a charge.