Inotify watch limit exhaustion from file explosion
Your task
A config-watching sidecar (Go) that hot-reloads files for a large fleet starts logging 'too many open files' and stops picking up config changes, though network requests still work fine. The process fd count is moderate (under 4k) and well below the 64k ulimit, yet adding a new watch fails. cat /proc/sys/fs/inotify/max_user_watches shows 8192, and the app added a recursive directory watcher last release that watches a tree that recently grew to ~9000 files after a config refactor split one big file into many. How do you triage and fix?
It covers
- Epoll
- Inotify watches
- File handles