Docker: Automatically remove containers that have been running too long
Posted by Admin • Thursday, October 20. 2016 • Category: DevOps, LinuxWhy Because my Jenkins setup sometimes starts containers and forgets about them. Either it thinks it failed to start one, or the container itself has trouble starting. Either way, I'm left with containers that are running, trying to connect to Jenkins in vain, forever. The proper way to fix this is probably to have the containers timeout at some point, but that mechanism is broken.
Anyway, the fix I have is a true hack: find containers that have been up more than 2 days and kill them. None of our jobs should run for more than about a day, so this is a safe limit. Here is a bash script to do this:
The only hard part here is converting a human-readable age from docker output (32 hours) to a number we can use for comparison. Fortunately, passing "32 hours" to "date -d" gives you a time 32 hours in the future, so from there it's easy.
#!/bin/bash
DOCKER_CMD="docker -H tcp://localhost:YOUR_PORT"
# max seconds before containers should be killed
KILLTHRESHOLD=${1:-172000} #default max age in seconds
if [ "$KILLTHRESHOLD" -gt 0 ] ; then
$DOCKER_CMD ps --format "{{.ID}}:{{.RunningFor}}" | while read line ; do
ID=$(echo $line | cut -d : -f 1)
HUMANAGE="$(echo $line | cut -d : -f 2)"
FAKETIMESTAMP=$(date -d "$HUMANAGE" +%s)
AGE=$(echo "$FAKETIMESTAMP - $(date +%s)" | bc)
# echo "Age is $AGE for $ID"
if [ "$AGE" -gt "$KILLTHRESHOLD" ] ; then
logger -t dockerwatch "Killing $ID, running for $HUMANAGE"
$DOCKER_CMD stop "$ID"
sleep 5
$DOCKER_CMD kill "$ID"
$DOCKER_CMD rm "$ID"
fi
done
fi
0 Comments
Add Comment