While cloud solutions certainly can provide ease of use and provide an easily integrated solution often they eventually turn up unexpected or unforeseen issues, such as GDPR and CCPA compliance or elevated costs.

There are other issues too that often aren’t considered up front, not having full access or control over the data stored or simply needing to process the data in a different way.

The Challenge: Collecting logs from a variety of sources and allowing them to be easily accessed

The project criteria were as following:

• Collect logs from physical servers, applications running in native environments and in docker containers.
• Store the logs in a centralized location where they could be processed easily for a shorter time.
• Store the logs in a long term storage for complying with regulatory requirements.
• Make use of proprietary infrastructure.
• Add an interface for querying of the logs.

Considerations

A variety of technologies were considered and of course there are several ways this can be solved. Syslog Remote and Forwarding, Vector, FluentBit to name a few. In this case it was expressed that the client wanted to use NewRelic for infrastructure and application monitoring and therefore it also made sense to use that as an additional short term storage. NewRelic has a very good query language and easy to use interface and it would avoid the need of setting up something else, such as Prometheus and Grafana.

The NewRelic agent, which is used to ingest metrics and default logging from servers, is based on FluentBit. FluentBit is very flexible and can be configured to collect data from many different sources, both locally and via remotes. The fact that a standalone FluentBit service can be run in parallel with the NewRelic agent allows for even more flexibility.

The Solution

FluentBit is a modern and stable solution that is far more flexible than most alternatives. This together with the wish to use NewRelic and that it’s definitely favorable to keeping the technology fragmentation as limited as possible made FluentBit the obvious choice.

It ticks all the boxes of this challenge:

• Takes input from a variety of sources. Required in this case:
  • Syslog
  • Forwarded logs from other FluentBit instances
  • Application logs
  • Docker container logs
  • Etc.
• Outputs to all required destinations:
  • FluentBit Remote
  • NewRelic
  • Long term network storage

To solve the short and long term propriety storage a FluentBit service was set up that accepted forwarded logs from all applications and servers, then multiple outputs writing to different destinations would ensure the storage.

Log rotation was applied on the short term storage to automatically manage the persistence period while the long term storage is kept indefinitely.

Parallel to this proprietary storage management the FluentBit services send logs and metrics to NewRelic via the NewRelic agent and therefore providing the additional Nice-to-have objective of using NewRelic as a data query and monitoring platform.

We can help!

This project is a good example of stable out-of-the-box solutions that we are able to create. We don’t just look for quick fixes; we engineer stable, efficient, scalable solutions that address both current needs and future flexibility.

If you are facing similar challenges, reach out to us. We specialize in unlocking your infrastructure’s potential, ensuring you can keep pace with the future.