CloudWatch Logs
CloudWatch is a service in AWS that allows us to monitor metrics for any other service in AWS. Here's their 2 minute marketing video:
That video doesn't go into any details, but it does list off a lot of features that cloudwatch is capable of.
Three big features of cloudwatch are:
- Logs: Collect and monitor logs from AWS resources, applications, and custom applications.
- Metrics: Collect and monitor metrics from AWS resources, applications, and custom applications.
- Events: Monitor and respond to changes in AWS resources and applications.
We will cover a lot more cloudwatch features later on, for now we're just going to focus on how to send logs to cloudwatch logs.
We will send logs from an application running inside an EC2 instance to cloudwatch logs. This means that we will be able to monitor and query the logs for an EC2 instance, without having to login to an instance with ssh just to view the logs.
We're only be scratching the surface of what CloudWatch Logs can do. But even just being able to easily access logs for a EC2 instances is a huge benefit! Any plan for monitoring an application should include easy access to the logs.
CloudWatch Logs Free Tier
Sending Logs to CloudWatch
To send logs to cloudwatch logs from an EC2 instance, we need to do the following:
- Configure the EC2 instance to run your application.
- Setup an IAM role for the EC2 instance.
- Attach the cloudwatch agent server policy to the IAM role.
- Install the cloudwatch agent on the EC2 instance.
- Configure the cloudwatch agent to send logs to cloudwatch logs.
We will be setting up the qr code web app EC2 instance, and have it send logs to cloudwatch logs.
Setup The App
Do this on the instance that you already setup in the previous part so that it can access the S3 bucket and cloudwatch logs.
Once you've set everything up, the application should be working just as before and you should be able to view the application logs by running:
Or to tail logs in real-time:
Remember that you can show logs for any individual service with the -u flag or all services without the -u flag.
Show timestamps
00:00
CloudWatch is a service in AWS that allows us to monitor our applications and infrastructure
00:06
with things like logs and metrics and it's powerful. There's a lot of things we can do
00:11
in CloudWatch but in this part we're just going to focus on setting up CloudWatch logs for an
00:17
application running on our EC2 instance so that we can have our app running and then we can view
00:22
all of the logs for it in a different service in AWS. And for the logs in the free tier we get
00:27
five gigabytes of data stored for free which is way beyond what we'll actually use so all of this
00:33
will be in the free tier and you can see more about the pricing at the CloudWatch pricing page
00:40
but we don't really need any of that for now. So the steps we're going to go through for this are
00:45
going to be we're going to configure the EC2 instance, set up an IAM role for that EC2 instance,
00:50
set up a policy that will allow the instance to do things to CloudWatch and we already did all of
00:56
this in the last part.
00:57
These last two things that we need to do to be able to have our instance send system logs to
01:04
CloudWatch. We're going to need to install the CloudWatch agent on EC2 and then we're going to
01:09
configure the CloudWatch agent to send the logs to CloudWatch. And for this we're going to be using
01:14
the QR code application that we set up in the simple web app section. So the very first step
01:19
here the thing that we need to do is actually go completely reset up that EC2 instance. So I am
01:26
going to speed my way through the process and I'm going to go ahead and set up the EC2 instance.
01:27
I'm going to speed my way through this right now but you can stop you can watch the video and go
01:31
through this in your own pace if you want. But the first thing we need to do is I'm going to go
01:35
into the Playground account and set up a new EC2 instance. And if you want to fast forward to when
01:40
I've set up the simple web app I'll just put the timestamp right here. So this is going to be my
01:45
QR app. It's going to be running on Ubuntu, T2 Micro. I'm going to use my existing key pair.
01:53
And for the network settings we're going to allow SSH, HTTPS and HTTP traffic.
01:57
From anywhere 8 gigabytes is good and we're going to launch this instance. Then when that's set up
02:02
I'm going to connect this using SSH. And then once I'm logged in I'm just going to go through
02:08
all of those instructions again. So I'll start with sudo app update, sudo app upgrade. And then
02:16
the first step is to install MySQL. So I'll just paste those commands in. Then we'll log into the
02:27
database. So I'm going to do that setup. So we can install bun for the web application. I'm actually
02:32
just going to download the source code directly onto the EC2 instance instead of downloading it
02:38
then copying it up to the instance. Which means I'll have to unzip this. And then I'll move
02:44
everything into app. So if I look, yep there's my entire application in the app directory. Then I
02:50
need to cd into app and bun install all of the dependencies. And then I'm going to migrate the
02:56
database. Then I'm going to install the database. And then I'm going to install the database.
02:57
Then I need to install caddy for the reverse proxy. Then we'll modify the caddy file and I'm
03:04
going to use this exactly as it is. So forward traffic to the localhost 3000 application which
03:12
is going to be the app that we set up soon. I'll just reload caddy. Then we need to set up
03:16
the app.emv file which is just going to have the MySQL URL in it for now. And then we'll change
03:23
the permissions for that file. Then we're going to set up a systemd
03:27
service file for the application. And here is the code for that. I think I can use it exactly as it
03:33
is because my application is in the app directory and I'm using bun to start it. I just need to
03:37
start that service. So now if I curl localhost, I should spell localhost correctly. There we go.
03:43
So my application is set up. I can curl localhost. I haven't set up a domain name for this. I'm not
03:48
actually need one just right now. So if I go back to this instance that is set up, I can grab the
03:55
public IP address.
03:57
Just right here. And if I visit this in a browser, I should be able to see my QR code web app. There
04:02
it is. Okay. So then I'm going to head back to the CloudWatch tutorial and we can see. Okay. So the
04:09
next step is we want to check the logs for this application, right? Because this is going to be
04:14
logging things to standard out which is what most things on the system will do. And if I want to
04:19
view these logs for debugging or just for analytics purposes, I can run sudo journalctl-u
04:27
and then the name of the service. So these are the logs that we're getting. They're all JSON logs and
04:32
there's not much to see there. If I actually wanted to tail these logs in real time, I can run it with
04:38
dash fu and now we can see logs come in from the application. So if I go back to the app and I don't
04:44
know, I think if I add an image that should cause it to log some stuff. Yeah, we can see new logs
04:50
coming in. I made a post request to slash API slash QR slash preview. So these are just standard
04:56
application logs and that's fine. If I log in, I can see that I can see that I can see that I can
04:57
log into this instance. I am able to run that command and see the logs coming from a specific
05:02
service. And I could do the same for other services running like I could do this for caddy.
05:06
I could do this for my SQL. But if we want to be able to view these logs without logging into the
05:10
instance, we can use something like cloud watch. And this isn't exactly a requirement right now.
05:14
We have a single application running on a single EC2 instance. But as soon as you're unable to log
05:20
into the instance, let's say we set this up on a private subnet and we can't just SSH onto it,
05:23
or we have many EC2 instances that are horizontally scaled and we need to be able
05:27
to view the logs for all of them and not just have to log into each of them to view the individual
05:30
logs, or we're using containers or Lambda functions, whatever it is. At some point,
05:34
it becomes necessary, pretty much a requirement to send these logs somewhere else so that we can
05:40
aggregate them and view them in a much more simple way. So what we want to do right now
05:45
is get these logs to go to cloud watch. So we don't have to log into the EC2 instance to view
05:49
all of these logs. So if we go back here, these are now the instructions. So there are full
05:54
instructions. I've included links within this tutorial. Uh,
05:57
if you want to view the entire instruction set from AWS, but I just have exactly what we need
06:03
for what we're doing right now in these steps. So I'm just going to copy this right here. So this
06:07
is going to download the cloud watch agent onto this Ubuntu instance. So I'll paste that in here
06:15
and I'm going to put these side by side. So this will download the agent and this is actually the
06:20
installer for the agent. So as soon as that is downloaded, we're going to need to run that
06:23
installer and actually set up the cloud watch agent on the instance. And this just makes it
06:27
really easy to gather logs and metrics from an EC2 instance and send them to cloud watch. And while
06:33
this is downloading, I actually just realized that I missed one of my own instructions. I was
06:37
supposed to do this on the instance that I already set up in the previous part. So that was the
06:41
instance that already had the IAM role and policy set up. So it would actually be able to access
06:45
cloud watch because right now this EC2 instance doesn't have an IAM role. It's going to be
06:50
unable to access S3 buckets or cloud watch. So what I'm going to do is go back to my EC2 instance.
06:57
And I'm going to modify this. So this is the current instance I have running. I already
07:01
terminated the one from the previous part. And I'm going to go to actions, security,
07:06
modify IAM role, and I still have the test IAM role set up. So I'm going to select that and I'm
07:12
going to update that IAM role. And if I go to the security tab, we can actually take a look at this
07:17
role. So I'm going to select the test IAM role and we should be able to see that this role has the
07:21
cloud watch agent server policy. And this is what's important. So we need a role assigned to this EC2
07:26
instance that has the cloud watch agent server policy. And that's going to allow the instance
07:30
to actually write logs and create logs groups within cloud watch. And this one also has the
07:35
IAM demo S3 policy that we made in the last part. And I don't need this right now. It's just this
07:41
cloud watch agent server policy. That's what's important for getting cloud watch to work.
07:44
So now that that has finished downloading, we can install the cloud watch agent on the
07:50
instance. And now that's installed, we're ready to start using cloud watch. So there is a wizard
07:56
that we could use and there's instructions here on how you could go through all the steps to set
08:02
up cloud watch in a more detailed configuration for a specific instance. But since we're just
08:08
setting this up to send logs to cloud watch, I find it a little bit simpler to just create the
08:12
configuration file from scratch. So what we're going to do is create the configuration file.
08:17
It's at slash up AWS, Amazon cloud watch agent bin config dot JSON. So there's going to be a
08:22
JSON file where we're going to put the settings for cloud watch. So cloud
08:26
watch is going to act based on the settings that we put in this file. And I am just going to copy
08:30
this entire JSON into here. So this is running as the cloud watch agent user, and we're telling it
08:36
to grab the logs from syslog. So this is where the default location is for the logs for every
08:43
application on the system. And that's also why it's important when I'm just going to go back to
08:48
the simple web app section here. When we set up our system D service, we gave this a syslog identifier.
08:56
This will show up as QR code app in syslogs in this location. And we're going to get cloud watch to
09:01
read the logs from this location and send them to cloud watch. So once we're in cloud watch,
09:06
we'll be able to identify this specific application that we have running this bun app
09:09
by syslog identifier right here. But I'm going to head back to the cloud watch log setup. So this
09:15
is where it's grabbing the logs from. We're going to create a log group name. And since this EC2
09:20
instance represents my QR code app, I'm just going to call it QR app dev. I'm just pretending
09:26
this is the developer instance and EC2 just to specify is for the entire EC2 instance. This is
09:31
just a custom name. And then the stream name is going to be for the current instance ID. So if I
09:36
had multiple EC2 instances running my application, I'd be able to identify it by its ID here. And
09:41
then we can have cloud watch keep track of the logs for a certain amount of time. I'm going to
09:44
have it persist them for 14 days. So they're going to auto delete after 14 days, which means that in
09:49
this account, we're not going to keep building up logs forever and go over that five gigabyte
09:53
free tier limit. So this is a pretty simple JSON
09:55
configuration and this will make sure that we get the logs sent to cloud watch. So I'm going to save
10:00
this file. Then I'm going to add the cloud watch agent to the system admin group, which allows it
10:07
to basically monitor everything for the system. And then we're going to run this command, which
10:13
will run the cloud watch agent using the configuration in that JSON file. And this should
10:19
immediately start sending logs to cloud watch. So it's going through the steps here. It's reading the
10:25
JSON configuration, validating it. And it looks like all of that succeeded, but we can double
10:32
check that. So if we check the status of the cloud watch agent, yep, it's active, it's running.
10:37
Everything seems to be good. Those logs that we saw coming from the application service and from
10:42
the system, we should now be able to see all these logs appearing in cloud watch. So I'm going to head
10:46
back over to the AWS dashboard and I'm going to open up the cloud watch dashboard. And in here,
10:54
we should see a section called the cloud watch dashboard. And in here, we should see a section
10:55
called log group. So this is where we're going to be able to view our logs. And here is that log
11:00
group name that I set up in that JSON configuration file. So if I click this, we should get all of the
11:05
different log streams. So like I said, if I had multiple EC2 instances, I would see all of their
11:09
IDs show up here. Just because I have one instance, I have one ID. So I'll click on that.
11:13
And then here are all the logs for the EC2 instance. I can see those all coming out and they
11:17
are for everything that is producing logs on the EC2 instance. So I can see logs for the cloud watch
11:25
agent. I can see them for my application. Those are these JSON logs here, the QR code app. I can
11:30
see the logs for caddy. So this is where I'm going to be able to monitor and query the logs for the
11:34
entire system. And if I click resume at the bottom, I should be able to see logs come in in
11:39
real time. So if I go back over to the app and I think if I click on these different selections
11:45
here, it should be creating logs for me that I can see. Yeah, start showing up at the bottom.
11:49
So here is one from my QR code app. And it looks like it was processing the QR code.
11:55
And I just had that in a console log on the server. So I can view all these logs here,
11:58
which is great. We can also head over to log insights here. And this is where we're able
12:04
to query the logs in much more detail. So I'm going to bring over the tutorial here so I can
12:10
just copy straight from there because there's this log insights query language that we're
12:16
using right now. And I'm not great at writing these queries. So I'm just going to copy and
12:19
paste these. So here we can get the timestamp of each log and the message of
12:25
the log sorted by timestamp and just limit that to the last 20 logs. So if I run this query,
12:30
I have to actually select the log group. So there's my log group. If I run this query,
12:33
I'll be able to see the logs for every single log within this QR app dev EC2 log group. And
12:40
then down here, we can see how these logs have been queried. So we got the timestamp and the
12:45
message here for the last 20 logs queried by timestamp, which kind of just looks like the
12:51
logs from before. And we can come through and we can see all of these. But then,
12:55
we can go a little bit further with this query language here. So instead of just querying
13:00
them normally, I could query where the error message contains the word error. So if I had
13:06
error logs where I clearly stated that this is an error and I run this query, I should
13:10
be able to see any of my error logs. It looks like I actually have two error logs. So that's
13:15
interesting because I have some code in my application that is checking if I have an
13:22
IAM role already set up. And I think I set up the app before I had the
13:24
role set up. So I did get that error earlier on. So that's pretty cool. I can see all the error
13:29
logs there and then I can monitor those a lot easier. And then I could even just query using
13:34
a little bit more advanced query, anything from the QR code app logs, those specific logs. So I
13:41
won't be getting logs from Caddy or MySQL. It's just for this application. And then I can pass
13:46
the message into its different components in a little bit of a nicer way. So I get
13:50
the JSON string appearing front and center. So here are all the logs that I have. And then I can
13:54
see all of the application logs just from my app and I can see exactly what's going on. So there
13:58
was a post request made and a bunch of other little messages being logged out here. So on
14:04
this dashboard, I can basically query down and get any logs that I want for anything that's
14:09
sending logs to CloudWatch. And logs insights query language is the traditional query language
14:15
that's been used. More recently, they have this open search SQL query language, which is similar,
14:20
but is more of a SQL type language. And I haven't used this much. So I'm
14:24
not super familiar with it. But this does look like a bit of a nicer way of querying in CloudWatch.
14:29
But I'm gonna head back to the QL over here. And we're going to run another log, which is just
14:35
going to show us the total number of get requests versus post requests here. So I'm going to count
14:41
the posts, I'm going to count the gets, just so I can get a sense of the ratio for this application
14:45
of how many requests are being run. And in this log right here, it doesn't look great,
14:49
but I can see the number of post requests seems to be seven, and the get request seems to be
14:54
empty. Maybe I should make some get requests. So I'm going to save this QR code, which should make
15:00
a post request, but then I think it makes a get request to then get all of the QR codes from the
15:04
API. So if I went to save codes, that should be a get request now. So if I go back to CloudWatch,
15:10
and get rid of that warning, if I run this query again, then we now have two get requests being
15:16
logged and nine post requests. But with kind of aggregate analytical queries like this,
15:20
we can also go to the visualization tab. And I'm going to
15:24
view this as a bar graph here. And now I can see pretty clearly the number of post requests versus
15:30
the number of get requests. And I'm just taking this from the logs coming from my server application,
15:36
it's just sending them to CloudWatch. So this video isn't an extensive deep dive into how to
15:40
use every single feature in CloudWatch, I just want you to get a sense of what CloudWatch is
15:45
here for, we can send logs to CloudWatch. And then we can just view them in that other page,
15:49
where we saw the log groups for the application. And this can be sufficient,
15:54
a lot of the time, you just come into the log stream, you can scroll through these logs,
15:57
you can even filter, I could type in the word error up here, and it would filter for those.
16:00
But then we can come into these other dashboards, we can query for specific logs using log insights,
16:06
QL or the open search SQL, we can even visualize those logs here. And this is even just scratching
16:11
the surface of CloudWatch, we can get all types of metrics and other things to CloudWatch. Like if I
16:17
go over to dashboards right now, this can cost money. But if you use the automatic dashboards,
16:23
those don't cost anything.
16:24
So if we go to the automatic dashboard for EC2, I can get a whole bunch of information and stats
16:28
about CPU usage, network usage. And we can also trigger things based on CloudWatch metrics. So I
16:35
could say, when my CPU usage hits a certain threshold, I want to spin up more EC2 instances.
16:40
So it's a really, really powerful service. And we're just kind of scratching the surface by
16:45
using it just to send our logs from EC2. But this is pretty much a requirement for most web
16:50
applications, we need to send our logs somewhere and be able to view them and query them. And
16:54
CloudWatch is the way we do that in AWS. And we can even integrate CloudWatch with other services.
16:59
So we could have CloudWatch logs funneled through Kinesis Firehose, which is another service in AWS,
17:05
send that to Lambda functions, which can then format the logs and store all of our logs in
17:10
S3 buckets, we can then query the logs in S3 buckets using Athena, which means that we could
17:15
have petabytes of analytical logs in S3, and run simple SQL queries to then get a bunch of data out
17:22
of that. And that can be really, really easy to do. And that's the way we do that in AWS.
17:23
It's really handy in bigger businesses when you need to make business decisions based on
17:28
unstructured data, like something you might get from your application logs. But that is way beyond
17:32
the scope of what we're doing right now. All we want to make sure is our applications are able to
17:36
send the logs to CloudWatch, which we can do right now. And then in the next part, we're going to
17:40
keep this instance running, because we're going to set up IAM to allow us to send images to S3 from
17:48
that QR app.