Unable to update Silo Entry

[matthewbrumpton-oddsbods]

We have a large number of KubeMembershipTable exceptions coming from our orleans application, the errors start after the application has been running for 5-9 hours.

Message: Unable to find silo entry 172.19.16.62-50000-374433332.

CategoryName: Orleans.Clustering.Kubernetes.KubeMembershipTable

Exception type: System.InvalidOperationException

Failed method: Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext

Problem Id: System.InvalidOperationException at Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext

Assembly: Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null

Call Stack: System.InvalidOperationException:
at Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext (Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)

[matthewbrumpton-oddsbods]

Another observation is OrleansClusterVersion is deleted after a period of time and then being recreated after we restart the app.

OrleansClusterVersion after we restart the app:

apiVersion: v1
items:
apiVersion: orleans.dot.net/v1
clusterId: clusterId
clusterVersion: 4
kind: OrleansClusterVersion
metadata:
creationTimestamp: "2021-11-15T14:43:13Z"
generation: 5
name: name
namespace: namspace
resourceVersion: "148545790"
uid: d4a66cf5-18a9-4474-bf88-351d496ac61d
kind: List
metadata:
resourceVersion: ""
selfLink: ""

OrleansClusterVersion when deleted:

kubectl get clusterversions --namespace namespace -o yaml
apiVersion: v1
items: []
kind: List
metadata:
resourceVersion: ""
selfLink: ""