Sometimes after a required host system reboot, existing Docker containers may be locked by AppArmor and show “Permission denied” error. Basically you lost control on your containers and you cannot stop/start any of your containers.
I won’t be able to explain how or why this happens but I do find a simple solution to unlock this with a less harmful way (hope so…).
Step 1: Safely Shut Down ALL Containers before System Reboot
This is an extremely important step for you to carry out before any host system reboot for whatever reason. If you don’t shut down containers properly before the system reboot, you will definitely encounter all kinds of issues, such as duplicated Docker networks, damaged inter-container connection, etc, and you will probably spend the next two days just to fix those issues in order to bring your containers back to life.
So please please remember to shut down all your containers before the system reboot, it just takes you two lines of command to shut down and check:
sudo docker-compose stop
docker psStep 2: Remove AppArmor Policies that Locked your Containers
AppArmor is a Linux Security Module implementation of name-based access controls. It confines individual programs to a set of listed files and posix 1003.1e draft capabilities. It is also installed and loaded by default since Ubuntu 8.04 LTS. You may encounter suggestions to uninstall AppArmor when searching for solutions to unlock the containers. But this is at the cost of your system’s security so I wouldn’t recommend that.
It seems that the containers are actually locked by some loaded profiles but are unknown to AppArmor. So what we do is to help AppArmor to clear the profiles that it does not know. It can be achieved by 2 simple lines of command:
sudo aa-status
sudo aa-remove-unknownaa-status will report various aspects of the current state of AppArmor confinement, including the running state of AppArmor itself, number of profiles that are loaded, in enforce mode or in complain mode, number of processes that are running, having profiles in enforce mode or in complain mode.
aa-remove-unknown will inventory all profiles in /etc/apparmor.d/ (in my case, the default list of profiles since I have not customized my ApprArmor before), compare that list to the profiles currently loaded into the kernel, and then remove all of the loaded profiles that were not found in /etc/apparmor.d/. It will also report the name of each profile that it removes on standard out.
For my case, these two lines just restore the AppArmor to its default list of profiles to load into kernel. After this, you should have regained your control over your containers:
As I have mentioned, I don’t know how or why the system reboot causes AppArmor to lock Docker containers but at least this method offers a simple yet less harmful way to regain your control of your containers. I believe the permanent solution is to configure the AppArmor correctly based on the services you are running but it will require some in-depth knowledge on how AppArmor works. Right now I am happy with my dirty and quick fix.