I will be showing the process of logging from the OS on an EC2 instance to CloudTrail, then sending an alert based on a specific action performed on the EC2 instance using multiple AWS services.
The first step is to install the CloudWatch agent on the EC2 instance. I will showing how to install the agent from the SSM GUI in AWS.
In order to manage EC2 instances using Amazons SSM, you must have the SSM Agent installed. The following Amazon AMI’s have the SSM Agent pre-installed:
- Ubuntu Server 18.04
- Ubuntu Server 16.04
- Amazon Linux 2
- Amazon Linux 2017.09 +
Installing CloudWatch Agent Using SSM:
Login to AWS and click the Services drop-down. In the search bar type “ssm”. Select Systems Manager.
On the left navigation panel, select Run Command.
Then select Run a Command
This will bring you to a form. Select AWS-ConfigureCloudWatch under the Command Document section.
In the Targets section, you will see all the available EC2 instances available. Select the desired EC2 instances on which you will install the CloudWatch agent.
The Action section will have Install selected by default. Just ensure Install is selected from the drop-down.
Type AmazonCloudWatchAgent in the Name text-box.
Select Run.
The summary page will appear showing the status of the command.
Before configuring the CloudWatch agent we need to ensure the EC2 instances selected above have permissions to write to SSM. This will ensure we have the capability of saving the config file into SSM.
Navigate to IAM.
Select Roles -> Create role
EC2 will be using this role.
Next: Permissions. Search and select AmazonSSMFullAccess and CloudWatchAgentServerPolicy. Next: Tags. Next: Review. Create role
Now lets attache the role to the EC2 instances. Navigate to EC2 -> Instances. Select all desired instances, Right-Click, Instance Settings, Attach/Replace IAM Role.
IAM role* select the created role -> click Apply
Now that the permission have been added. Lets ensure the agent installed by logging into the EC2 instance and checking to see if the directories have been created.
cd /opt/aws/amazon-cloudwatch-agent
Next, create a CloudWatch agent configuration file by running the following command:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
Default/Default/No/Default/Default/No/No/Default/Default/Default
When prompted for “Do you want to monitor any log files?” Select Yes.
/var/log/secure
Any flat file directories can be configured, however for our purposes we will only configure the security logs.
Select Yes when prompted “Do you want to store the config in the SSM parameter store?”
Sending the config to the SSM paramenter store will allow the configuration to be sent to all EC2 instances in the environment.
Navigate to CloudWatch -> Logs -> Secure -> Unique Number
Now we see raw logs from our EC2 instance. We can now use this data to create Rules.
Lets alert on logins to EC2. Navigate to CloudWatch -> Logs -> select the radio button for the new log source and select Create Metric Filter.
We have a few options to define the filter pattern, however I will simply be using the match log event using quotes.
- Match log event that contain a term: “Accepted publickey for ec2-user”
Select Test Pattern and ensure the Filter Pattern matches the intended logs. Select Assign Metric.
Enter Metric Name. Create Filter.
Now we can select Create Alarm from the filter created page.
Enter the Name and Description of the new Alarm. Alert on Login is: >= 1 for 1 out of 1 datapoints. Treat missing data as: good. Whenever this alarm: State is ALARM. Send notification to: (Select New list)
Enter List name and target email for Alarm. Confirm the email. Create Alarm.
We now see the state is INSUFFICIENT DATA on the alarm.
After a few minutes, the state will change to OK.
Now lets trigger the Alarm by logging into the EC2 instance.
Log in CloudWatch.
Alarm history
Email delivered
Email contents
Great! Everything worked as expected. Now all these configurations are very scalable and have the ability to be applied to thousands of EC2 instances in your environment.
Feel free to reach out here if you have an questions or comments, and until next time, stay safe in the Trenches of IT!