Why is it so important now?

To answer this let's understand the transition(or the transformation) of how the application architecture evolved from monolithic -> distributed -> containerized

monolithic monitoring endpoints:

• Infrastructure- limited to a single machine

• Network - client to server network connectivity, throughput, and latency

• Application - a couple of application endpoints

distributed monitoring endpoints

• Infrastructure - *multiple hosts connected together to form a system, additional monitoring points(over monolithic) but similar check (CPU, memory, etc.)

• Network - *apart from what was already monitored in monolithic architecture additional connection monitoring between many hosts of the system

• Application - *Distributed applications across multiple hosts, multiple instances of applications working together, monitoring of parent applications, and the interaction between these applications on multiple hosts. Challenging to design and operate a scaled system

containerized monitoring endpoints

• Infrastructure - *Host infrastructure(similar monitoring endpoints like monolithic and distributed), additional monitoring of cluster(K8s, Docker swarm, Openshift) infrastructure, and finally container/s health( CPU, memory, etc.)

• Networking

  • Connectivity between microservices -

    • Containerization or microservices-based architecture with every microservice connected to every other service, auto-scaling, exponentially(need a better term to describe the scale) exploded the number of connections with the system. Apart from measuring the health of external connections, it became imperative to build inter-service connection health monitoring capability while developing containerized applications

  • Tracing the API interaction between microservices -

    • Another aspect that floated on the surface was tracing the API interaction flow between different microservices. This was the same challenge with distributed and one of the main reasons why engineers dreaded troubleshooting a scaled system. There were no tools available to identify the weak links and with microservices-based architecture, it is not possible to troubleshoot manually(period)

• Application -

  • More or less the same checks as for monolithic and distributed applications however now service orchestrators and load balancers are responsible for application and container health monitoring. Although most of it is abstracted from end-user and system admin and developers will build them in the development phase

You can now appreciate(the evolution) and anticipate the complexity of monitoring a contemporary application(and the associated infrastructure) however the "good" news is

there are many 3rd party open source tools available to monitor this architecture (which was missing during distributed design era) enabling the community to adopt and transform their applications