TE Controller disk gets full within days due to huge data getting filled up in postgres DB

[sanathpholla]

Describe the bug

TE Controller disk gets full within days due to huge data getting filled up in postgres DB

root@Client-te-1:/var/lib/postgresql/11/main/base# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          15G   14G     0 100% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/sda1        15G   14G     0 100% /te_host
root@Client-te-1:/var/lib/postgresql/11/main# du -h --max-depth=1 .
8.4G    ./base

Once the disk is 100% full,

• None of the get_ses_metrics/get_vip_metrics would return expected result
• clear_config call would also fail

te_dp's however continues to send traffic to VIPs without any issues seen

Only workaround is to redeploy TE Controller.

Reproduction steps

1. Start high scale traffic to 5K VS VIPs
2. Wait for around 1-2 days for the postgres DB to be populated with metrics

Expected behavior

Postgres metrics rotation should typically be done to prevent disk getting full

Additional context

No response