Deadlock on concurrent Connection.Dispose() and AsyncSubscription.Start()

[paul-barry-delongchamps]

Observed behavior

We have had multiple occurrences of this scenario:

Thread#1
AsyncSubscription.Start()
  lock Subscription.mu
  Connection.sendSubscriptionMessage(this);
    lock Connection.mu

Thread#2
Connection.Dispose()
  Connection.Close()
    lock Connection.mu
    foreach Subscription in subs.Values
      AsyncSubscription.close()
        Subscription.close(ownsChannel)
          lock Subscription.mu

Expected behavior

It should not deadlock

Server and client version

Client NuGet version: 1.1.8

Steps to reproduce

It usually deadlock on the second try:

static void Main(string[] args)
{
    while(true)
    {
        TryDeadlockNatsClient();
        Console.WriteLine("Didn't deadlock, retrying...");
    }
}

private static void TryDeadlockNatsClient()
{
    var options = ConnectionFactory.GetDefaultOptions();
    // setup options
    var connectionFactory = new ConnectionFactory();
    var connection = connectionFactory.CreateConnection(options);
    var subscription = connection.SubscribeAsync("subject2");

    var disposeTask = Task.Run(connection.Dispose);
    var subscriptionTask = Task.Run(subscription.Start);
    Task.WhenAll(disposeTask, subscriptionTask).Wait();
}