Log archival with SnappyFlow


Need for archival

Servers and applications generate logs to report informational events, error events, debug events or fatal events. These logs are used to troubleshoot problems, identify user behavior, identify hosts requesting access to the applications, identify anomalous behavior etc. Further, they are also used to analyze differences in behavior of an application on a day to day basis. Appearance of a log which was not seen earlier may indicate (a) a new scenario being triggered or (b) an unexpected behavior due to a change within an application.

In some scenarios, enterprise may need to analyze old logs for security forensics and gather crucial information about cyber-attacks, fraudulent access etc. To perform such analysis, logs will need to be retained for a longer period of time. Certain regulatory compliance also may necessitate longer retention periods for logs.

Existing solutions and gaps

Existing log management solutions like Splunk and Elasticsearch replicate logs by compressing and shifting logs from primary storage to an offline archive. When archived logs are required to be analyzed, these are brought back from the archival store to primary store and searched. This design has several shortcomings, namely:

  • It is not possible to search archive and consequently, users retain logs for longer periods in primary storage itself
  • The process of bringing back an archive to primary storage cumbersome, impacts the performance of the cluster as well as makes sizing of primary storage unpredictable
  • Searching archived logs usually spans a long period and involves a large amount of data. The optimum way to search archives is through background jobs and with an ability to retain results, search within results, setup recurring jobs etc. Both Splunk and Elasticsearch, only support interactive searches which leads to poorer usability as well as requires larger clusters

SnappyFlow’s Log Archival Approach

SnappyFlow’s SaaS solution provides comprehensive log archival functionality that leverages S3 compatible object store. Unlike competitive solutions users can search & visualize logs seamlessly in both the active data and archive data without the need to move data from archives to primary storage. Further, SnappyFlow analyses logs, extracts signature, adds search metadata and finally compresses the logs before storing the logs into archive.

Log Archival

Key benefits from SnappyFlow’s Archival feature to users:

  • 10 - 40x compression plus maintaining data in S3 translates to significant cost reduction
  • Extensive search using regex patterns
  • Ability to visualize results, zoom into results, search within results, join results, store results for reference
  • Achieve faster search of logs at petabyte scale as a result of superior organization, meta-data, signatures and other innovative technique
  • Signature based log filtering and analysis

Get in touch

Or fill the form below, we will get back!


Is SnappyFlow right for you ?

Subscribe to our newsletter