How to Check the Status of Internal Services

Securing web services in private networks comes with different challenges. In this blog post I would like to show how you can easily monitor your private web services located in a VPC with public uptime monitoring services such as Datadog, New Relic, Statuscake, Pingdom etc.

Internal services are closed to public communication. Monitoring the health status of these services, allows us to see if there is a problem, and interfere at the right time. At this point, an idea appeared in my mind. Why wouldn’t I proxy only health check requests from a specific requester with a small lambda function? So I decided to develop a Lambda function in Python and create and deploy a REST API with API Gateway in front of it to give this control.

As a first step, I will create a Lambda function via the AWS Console, Lambda service.

I set a name for the function and runtime that will be used. I chose Python 3.6 because I will develop a function with Python.

The next step is where we need to pay attention. Since the web services are located in private subnets, placing the lambda function in the same VPC is recommended otherwise the function cannot access these services. It’s also possible to interconnect VPC’s together with VPC Peering or Transit Gateway but I want to keep things simple OK? :) Let’s place our function in the same VPC with the web services that we would like to monitor.

The urllib3 library is sufficient for our task to create the lambda function.

I will map different services by different unique variables. The function will send a GET request to the desired service’s health check URL depending on the variable value. Returning the response of this health check URL will tell us if the services are down or up.

botocore.vendored is primarily used when including libraries for the Lambda function.

import json
from botocore.vendored import urllib3

http = urllib3.PoolManager()

def lambda_handler(event, context):
    # 1. Parse out query string params
    url = event['queryStringParameters']['url']
    r = http.request('GET', url)

    # 2. Construct the body of the response object
    transaction_response = {
        'url': url,
        'message': r.status
    }

    # 3. Construct http response object
    response_object = {
        'headers': {
            'Content-Type': 'application/json'
        },
        'statusCode': r.status,
        'body': json.dumps(transaction_response)
    }

    # 4. Return the response object
    return response_object

You can find the full code on GitHub!

Now, it's time to create the API Gateway!

I’ve created the API Gateway by following the wizard. Just choose REST as API type, name it, put some description in case of for a future reminder, and set endpoint type to Regional.

Now, we will create a resource and a method (GET) in this resource.

Proceed by setting a name to the resource and you can fill in the explanations below if you wish.

After the resource is created, the Create Method is selected from the Actions section and the Method creation section is started.

When you choose the GET method and proceed, the wizard screen will appear, which you will actually match the method with your Lambda function.

The installation will be completed by choosing the region and function where the Lambda is located.

You've come to the end!

Following the completion of installations, you can deploy the application from the Actions section. Select [New Stage] on your first deployment and give it a name as I did, like “test” or “production”. For next deployments for the same stage, you should select the existing one that you want from the dropdown menu. Then deploy your healthchecker API!

If you click on the stage name from the left sidebar, the url of the API will appear on “Invoke URL”.

Copy and paste this url to your browser and add the internal (or external) site that you want to check at the end of the endpoint as a get parameter, like:

https://?url=

To make sure that only health check requests are routed to your application, you can create a Web Application Firewall rule according to your health check endpoint by only allowing that and denying the rest of requests.

You can also add an IP restriction to your allow rule if your uptime monitoring service is providing you list of IP addresses that health checks will come to your end:

• https://docs.datadoghq.com/api/latest/ip-ranges/

• https://docs.newrelic.com/docs/using-new-relic/cross-product-functions/install-configure/networks/

For more information, you can check following documentation:

• https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-aws-waf.html