For the past few years, I have been using Amazon Web Services for the underlying infrastructure in several projects. Many of its services make my life much easier and allow me to focus on the quality of my products.  But when it comes to cronjobs, there’s nothing out of the box that allows you to manage your tasks.

I was looking into a way of avoiding having to set up and manage a cron replacement system (e.g. chronos), and easily set up tasks from AWS console itself. Here’s how I brought up Amazon’s Data Pipeline and their, recently released, EC2 Container Service together to create a cron service.

1. Create an ECS Cluster
A great advantage of using ECS for running your jobs is having them completely isolated from each other whilst using the same instances. Updating your jobs also means updating your Docker container… and nothing else.
You’ll need to set up a cluster and register, at least, one instance. This cluster will be used to run all of your jobs. For testing, You may simply use the busybox example.
Now the exciting part, running your job:

aws ecs run-task --region eu-west-1 --task-definition busybox --count 1

2. Create new pipeline scheduler
Now, we need a way of scheduling the tasks. Thankfully, Amazon Data Pipeline offers exactly that. It is meant for moving and processing data, but its scheduler and ability to run any AWS CLI command makes it a perfect fit for what we’re looking for.
Create a new pipeline, name it something obvious (e.g. Cron: Busybox), and under Build using template choose the option Run AWS CLI command. It will ask to choose what command you want to run, we’ll use the same as mentioned before:
aws ecs run-task --region eu-west-1 --task-definition busybox --count 1
Define when and how often you want your task to run (e.g. Run every 15 minutes), and configure everything else as it seems fit… you’re ready to go! (Note: you might need to define a new role that allows the instance to run ecs:runTask)

Once you activate your data pipeline, it should initialize a new instance and run the command every 15 minutes (or whatever you have defined). If you wish to change how often it runs, simply update the pipeline. If you want to update the job itself, simply update your docker container!

Bear in mind that this services used are not specific for cron jobs, so it might not offer advanced features like Chronos or Agenda. It should fit simple use cases though.

Update:
You can easily update this method to run a Lambda function and save on having an always-on cluster.

@AlexCorreia

Advertisements