Introduction

Monitoring is a strategic choice that could enhance the value being added to complex solutions. Complex applications are applications that consume and rely on third-party APIs, exchange large amounts of data or the ones that scale on-demand where the demand is unpredictable.

Monitoring can be implemented before the implementation phase has started, during implementation or integrated after the application has been deployed to production.

Questions that Every Manager Should Ask Before Planning to Implement a New Monitoring Strategy

Here is a list of questions that every manager should ask before aiming to integrate monitoring into the solution. These questions are not written in a particular order so each manager should reorder them depending on the project priorities.

• What areas of the system in question are you aiming to monitor?
• Why the selected areas are significant to the business?
• Is the goal to monitor these components at a particular pattern of time or constantly?
• Is troubleshooting technical issues can be done through monitoring?
• Which metrics and values would be sufficient to achieve the targeted monitoring level?
• How do you plan to generate reports from the monitored system and output data?
• Are you planning to take actions manually or in an automated fashion when monitoring exposes warnings and errors that require attention from you?
• Do you have a list of people who should notify when such an event occurs?
• What medium do you need to use to notify those who should be concerned about such issues?
• Do you need to automate the response to incidents discovered from monitoring?
• How monitoring can help triage issues and incidents into technical for support and non-technical for stakeholders?
• Are there any constraints that limit your ability to implement a coherent monitoring plan and strategy?
• Can third-party libraries send monitoring data and logs to your Azure subscription?
• Are all integrated components compatible with the versions of dependencies?
• Does everyone in your team understand the definition of System availability and Health Status?
• Can monitoring mitigate the potential security risks that may be introduced by third-party libraries and other code vulnerabilities?
• Are you following best practices? Is monitoring going otherwise to add overhead?
• How much increase in cost will monitoring introduce?

Metrics and Logs

Azure automatically generates logs for deployed artefacts. These logs include Create, Update, and Delete operations for both the subscription-level and tenant-level. In addition, Azure generates Data Plane Logs which are events logs that are generated from running services such as the events generated from Virtual Machines OS and Applications Gateways and other services. In nutshell, Azure distinguishes between two different types of monitoring Metrics and Logs. Metrics are time-series data that azure services auto-generate to reflect the status of the services and their performance. Logs are events that can be generated when the services trigger a new event such as when Application Insights, for example, captures exceptions in the deployed application’s code execution path. These logs are stored in Azure storage. Therefore, there are maybe additional costs involved in storing large amounts of log files. The following image shows both Metrics and Log items under the Azure Metric service.

the image below shows ingress/egress for the storage account that I have recently created. These metrics data are auto-generated via the Azure portal. The chart visualises the data using Avg aggregation method.

Full Stack Application Monitoring

Azure portal uses Kusto Query Language (KQL) to query logs data. The language supports filtering, retrieving, searching, aggregating, and visualising log data.

The following example is for CPU usage trends over the last day.

With monitoring you can achieve all of the following:

• Monitoring resources activity and performance over a period of time
• Monitoring systems’ and components’ health
• Monitoring system traffic for security analysis
• Monitoring for cost optimisation
• Monitoring for system troubleshooting
• Monitoring for reporting and visualising sub-systems interactivity
• Monitoring for Auditing and compliance

The following diagram shows Azure monitoring landscape

This diagram gives you an overview of all the tools that you can use with Azure to achieve the following

• Collect logs from sources
• visualise and generate reports either in the portal or using external tools such as PowerBi and Grafana
• Analyse output
• Create alerts and notifications or take actions based on the outcome of the analysis
• Integrate monitoring data with other components or receive additional input from external sources using Monitor API

Full Stack Application Example

If your application uses the following components:

• Virtual Machines
• App Service
• Azure SQL DB
• Azure Storage
• Application Gateway
• Azure Loadbalancer
• NSG
• AKS
• Network Connection
• Azure Traffic Manager

Then you need to consider the following attributes to be included in your monitoring plan:

• Performance metrics – CPU, Memory and Disk usage
• Disk I/O for SQL IaaS VM
• Guest OS Diagnostic data
• Average response time
• Data in and Data out
• Http server errors
• CPU percentage
• Memory percentage
• Basic metrics
• Errors
• QueryStoreRuntimeStatistics
• QueryStoreWaitStatistics
• UsedCapacity
• ResponseType: AuthorizationError
• ResponseType: ServerBusyError
• ResponseType: ServerTimeoutError
• Metrics: Failed Requests
• Metrics: Throughput
• Metrics: Healthy Host Count
• Performance log monitoring
• Firewall log monitoring
• Data path availability
• Health probe status
• NSG low log in Network watcher
• Health status
• Performance: Node CPU & Memory utilization
• Container performance monitoring through Azure Monitor for containers
• Monitor connection to a dependent component IP address through
• connection monitor of network watcher
• Queries by endpoint returned
• Endpoint status by EndPoint

Conclusion

Monitoring Azure components can add significant value to your application and business. However, there are some considerations that you need to address in advance. Whilst it’s possible to integrate monitoring after the application has been deployed to production, it’s recommended that this step is considered ahead of time.

If you have any questions please let me know.

You can find more information about Azure monitoring in the book Understanding Azure Monitoring