AWS Solutions Architect: Logs & Monitoring

7 | Written on Sun 19 April 2020. Posted in Nuggets | Richard Walker

CloudTrail, CloudWatch, and AWS Config are three AWS services that can help with the health, performance and security of AWS resources and applications.

Tracking Performance

Understanding how AWS resources are performing can tell is they're powerful enough to handle load and if scaling is necessary.

Detecting Application Problems

Some application problems may be latent or hidden and go unreported. Checking logs for warnings or errors can alert to such issues.

Detecting Security Problems

Users having excessive permissions and accessing resources they're not supposed to, pose a security risk.

Logging Events

Maintaining a log of every single action that occurs against AWS resources is crucial for troubleshooting and security investigations.

Maintains an Inventory of AWS Resources

Maintaining an up-to-date inventory for existing resources, how they're configured, and their relationships and dependencies can help understand the impact of changes and ensure compliance is met.

  • CloudTrail keeps detailed logs of every action that occurs against AWS resources, providing the WHo, What and When information.
  • CloudWatch collects metrics from AWS ad non-AWS resources such as on-prem servers. It collects logs and provides alarms for notification or automation.
  • AWS Config tracks how AWS resources are configured and change over time.

CloudTrail

An event is a record of an of an action that a principle performs against an AWS resource. CloudTrail gives you a record which includes the action, resource and its region, who performed it and when. Both API and non-API actions are logged regardless of if performed via web console, SDK or CLI. Events are classified into management events and data event.

Management Events

Management events include operations that a principle (a principal is a person or application that can make a request for an action or operation on an AWS resource) executes against an AWS resource. Management events can also be known as control place operations. Management events are further grouped into write-only and read-only events. Logging in or creating EC2 instances are examples of write-only events, only successful events are logged. Read-only events include PAI operations that read resources but can't make changes.

Data Events

Data events track two types of data plane operations that tend to be high volume.

Event History

CloudTrail logs 90 days of management events by default and stores then in a viewable. searchable and downloadable database called the event history on a regional basis. S

Trails

A trail is a JSON configuration that records specified events and deliverers them to an S3 bucket.

Log File Integrity

With log file integrity validation enabled, every log file is hashed, making it easy to detect when a file has been modified.

CloudWatch

CloudWatch functions as a metric repository that enables the collection, retrieval and graphing of numeric performance metrics from AWS and non-AWS resources. All AWS resources automatically send their metrics to CloudWatch. Optionally, you can send custom metrics from your applications and on-prem servers.

CloudWatch Metrics

CloudWatch organises metrics into namespaces. Metrics from AWS services are stored in AWS namespaces and use the format AWS/service. Metrics only exist in the region which they were created. A metric functions as a variable and contains a time ordered set of data points, each contains a time-stamp, a value, and optionally a unit of measure.

Basic and Detailed Monitoring

The frequency that an AWS service sends metrics to CLoudWatch depends on the monitoring type the service uses. Most support basic monitoring, some support both basic and detailed. Basic sends metrics every five minutes, detailed sends metrics every minute.

Regular and High-Res Metrics

Metrics with a resolution of less than one minute are high-resolution metrics.

Expiration

You can't delete metrics in CloudWatch, they expire automatically. Metrics get aggregated from hight-res to lower-res metrics. A High-Res metric is stored for three hours, then all data points from each minute-log period are aggregated into a single one minute data point. After 15 days, one minutes data points get aggregated into five minute data points and retained for 63 days. After that, they are aggregated into 1-hour resolution metrics and retained for 15 months, after that they are deleted.

Graphing

Metrics can be visualised by graphing data points over time.

CloudWatch Logs

CloudWatch Logs is a feature of CloudWatch that collects logs from AWS and non-AWS sources, stores them, and lets you search and extract custom metrics. Some common uses include receiving CLoudTrail Logs, collecting application logs from an instance and logging Route 53 DNS queries.

Log Streams and Log Groups

CloudWatch Logs stores log events that are records of activity recorded by an application or AWS resource. Events from the same source are stored in a log stream. Log streams are organised into log groups.

CloudWatch Agent

The CloudWatch Agent collects logs from EC2 instances and on-prem servers running Linux or Windows. The agent can collect performance metrics. Metrics generated by the agent are custom metrics and are stored in a custom name-space that you specify.

Sending CloudTrail Logs to CloudWatch Logs

CloudTrail can be configured to send trail logs to a CLoudWatch Logs log stream.

CloudWatch Alarms

A CloudWatch Alarm watches over a single metric and performs an action based on its value over a period of time. The action can include things such as sending an email notification, rebooting an instance or triggering AutoScaling.

AWS Config

AWS Config tracks how your AWS resources are configured at a point in time. You can see what a resource configurations looked like at some point in the past versus what it looks like now.

Summary

You must configure CloudWatch and AWS Config before they can begin monitoring your resources. CLoudTrail automatically logs the last 90 days of management events even if you don't configure it. It's therefore a good idea to configure these services early on in your AWS deployment. CloudWatch, CLoudTrail, and AWS COnfig serve different purposes, and it's important to know the differences among them and when each is appropriate for a given use case. CloudWatch tracks performance metrics and can take some action is response to those metrics. It can also collect and consolidate logs from multiple sources for storage and searching, as well as extract metrics from them. CloudTrial keeps a detailed record of activities performed on an AWS account for security and auditing purposes. You can choose to log read-only or write-only management or data events. AWS Config records resource configurations and relationships past, present, and future. You can look back in time to see how a resource was configured at any point. AWS COnfig can also compare current resource configurations against rules to ensure that you're in compliance with whatever baseline defined.

Configure the different features of CloudWatch

CloudWatch receives and stores performance metrics from various AWS services. Custom metrics can be sent to CloudWatch and alarms can be configured to take one or more actions based on a metric. CloudWatch Logs receives and stores logs from various resources and makes them searchable.

Difference between CloudTrail and AWS Config

CloudTrail tracks events, while AWS Config tracks how those events ultimately affect the configuration of a resource. AWS Config organises configuration states and changes by resource, rather then by event.

CloudWatch Logs integration with CloudTrial

CloudTrail can send trail logs to CLoudWatch Logs for storage, searching and metric extraction.

SNS

CloudWatch and AWS Config send notifications to an Amazon SNS topic. The SNS topic passes these notifications on to a subscriber, which consists of a protocol and endpoint. SNS supports protocols such as http, https, email, email-json, sms, sqs and lambda.

Disclaimer

Information on this page was obtained from source: AWS Certified Solutions Architect Second Edition ISBN 978-1-119-50421-4

Notes taken are kept brief and for personal reference. I urge and highly recommend anyone using this page as a source of information to purchase the source material for the complete information. The original book is fantastic and includes exercises, practice questions, verbose explanations and extra learning resources.

COMMENTS