Modern environments become more and more complex every year. When many applications and services collaborate together to perform a single task finding a cause of a problem is similar to looking for a needle in a haystack. Good tools are needed to help. There are some that do a very good job of collecting logs, alerts or notifications but they focus on a specific problem and not on the problem space as a whole. Collecting just logs, alerts or statistical data is not enough. There needs to be a way to combine the data together and let it speak, so that data from many different applications can be correlated from end-to-end, and from high to low levels. ViaQ is a new project that aims at creating a framework for connecting data aggregation, processing, and analytic technologies that already exist into a coherent and flexible solution adaptable to multiple use cases.
There are some efforts that we want to leverage:
- OpenShift has begun shipping an EFK stack as containers - we want to leverage this work to provide our solution as containers, but perhaps not dependent on OpenShift
- There has been a lot of investigation of collecting event data such as logs using a message bus and feeding that data into analysis tools such as Apache Storm and Apache Spark - we would like to use a message bus based approach so that we can not only feed data to an EFK stack but at the same time feed data to an analytics tool, data warehouses, or any other application requiring a live stream of data
- There has been a lot of work done to describe a common data format so that logs from OpenStack (all of the various components and log formats if different from oslo logging), Ceph/Gluster, and syslog can be correlated together (e.g. timestamps, hostnames, node identifiers, etc.)
- Use the new CentOS infrastructure to build upstream images based on CentOS, use the CentOS CI, and eventually use the CentOS container image build and repository systems