vault 1.5.0 integration errors

[hgarrow]

Hi - This is my command history from a session trying to use the 1password vault plugin to compile and install it in my Vault 1.5.0 cluster on CentOS 7. The errors indicate that the plugin is incompatible with this version Vault.

$ go build -o vault/plugins/op-connect .

$ cp vault/plugins/op-connect /path/to/vault_plugins/op-connect

$ systemctl reload vault

$ vault plugin register -sha256=sha256sum /path/to/vault_plugins/op-connect |cut -d ' ' -f1 secret op-connect
Success! Registered plugin: op-connect

$ vault secrets enable --plugin-name='op-connect' --path="op2" plugin
Success! Enabled the op-connect secrets engine at: op2/

$ vault write op2/config @op-connect-config.json
Error writing data to op2/config: Error making API request.

URL: PUT https://myvaultserver.domain.com:8200/v1/op2/config
Code: 500. Errors:

• 1 error occurred:
  * internal error

$ vault secrets disable op2/
Error disabling secrets engine at op2/: Error making API request.

URL: DELETE https://myvaultserver.domain.com:8200/v1/sys/mounts/op2
Code: 400. Errors:

• Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

[jillianwilson]

Hi there! I've tried setting up the connect plugin with version 1.5.0 of Vault and am unable to replicate your issue. Can you provide any logs that the vault server may be outputting?

[hgarrow]

I have set up a single node vault instance and the plugin is operating correctly. My earlier install was a vault raft 3-node cluster. I will need to stand up another one of those to test if the problem was there or in something else from the initial build.

[SimonBarendse]

Hi @hgarrow, were you able to reproduce the issue? I'm interested to learn if there's more we should do.

If you are still running in to this error - do you have access to the server logs and could you share them with us? Hopefully this will be of help to find the cause of this error.

[NewsBankTechOps]

I haven't been able to get the docker container with the bridge functional to move further on testing, but from responses received so far, the single node Vault was successful in reproducing the same errors as a direct URI call, while the raft build has the behavior as shown above.

The fact that the plugin can be unregistered successfully in the single node instance makes me believe that it is the plugin integration with Raft that is missing something.

[hgarrow]

At debug logging in vault, This is all that I see after enabling the plugin. I also am unable to write the config file, and this is the version plugin that is working on the single node test vault server.

Jun 21 08:40:26 testraft vault[29373]: 2021-06-21T08:40:26.468-0400 [ERROR] secrets.op-connect.op-connect_9e9678de.op-connect.op-connect: plugin tls init: error="error during token unwrap request: Error making API request.
Jun 21 08:40:26 testraft vault[29373]: URL: PUT https://testraft.example.com:8200/v1/sys/wrapping/unwrap
Jun 21 08:40:26 testraft vault[29373]: Code: 403. Errors:
Jun 21 08:40:26 testraft vault[29373]: * permission denied" timestamp=2021-06-21T08:40:26.468-0400
Jun 21 08:40:26 testraft vault[29373]: 2021-06-21T08:40:26.488-0400 [ERROR] rollback: error rolling back: path=op/ error="Unrecognized remote plugin message:
Jun 21 08:40:26 testraft vault[29373]: This usually means that the plugin is either invalid or simply
Jun 21 08:40:26 testraft vault[29373]: needs to be recompiled to support the latest protocol."
Jun 21 08:40:43 testraft vault[29373]: 2021-06-21T08:40:43.570-0400 [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/h691a017d94e36d4fd9bd08666dcf3654b0952a014860c4e22bca844933bee047
Jun 21 08:41:26 testraft vault[29373]: 2021-06-21T08:41:26.106-0400 [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/hea25b4b95eb782ad0db1459fd062862146acdfd12db434a81e5d3eda1a7a6f19
Jun 21 08:41:26 testraft vault[29373]: 2021-06-21T08:41:26.456-0400 [ERROR] secrets.op-connect.op-connect_9e9678de.op-connect.op-connect: plugin tls init: error="error during token unwrap request: Error making API request.
Jun 21 08:41:26 testraft vault[29373]: URL: PUT https://testraft.example.com:8200/v1/sys/wrapping/unwrap
Jun 21 08:41:26 testraft vault[29373]: Code: 403. Errors:
Jun 21 08:41:26 testraft vault[29373]: * permission denied" timestamp=2021-06-21T08:41:26.455-0400
Jun 21 08:41:26 testraft vault[29373]: 2021-06-21T08:41:26.473-0400 [ERROR] rollback: error rolling back: path=op/ error="Unrecognized remote plugin message:
Jun 21 08:41:26 testraft vault[29373]: This usually means that the plugin is either invalid or simply
Jun 21 08:41:26 testraft vault[29373]: needs to be recompiled to support the latest protocol."

[jillianwilson]

I've spent some time trying to replicate this issue by recreating the environment and vault settings you've described but have not had much luck. I did manage to get a similar error when I misconfigured one of my test vault's api address field. Do you mind sharing your Vault configuration so we can rule that out as a potential issue? Could you also share your build sha and what branch you are building the plugin from?

[hgarrow]

This was built from v1.0.0 branch
https://github.com/1Password/vault-plugin-secrets-onepassword

vault version 1.7.1-1
golang from epel version 1.15.5-1.el7

sha256sum /etc/vault.d/vault_plugins/op-connect |cut -d ' ' -f1
c7feaa260b5724fce0032b8a61a7dc061f51e1e000426549610ada1b582475ce

/etc/vault.d/op-connect-config.json

{
    "op_connect_host": "http://OnePa-LoadB-....elb.us-east-1.amazonaws.com:8080",
    "op_connect_token": "reallylongprivatestring"
}

Key to the identities
standalone.example.com - autounseal transit vault instance
first.example.com - first built raft node
second.example.com - second built raft node
third.example.com - third built raft node

/etc/vault.d/vault.hcl

ui = true
api_addr = "https://first.example.com:8200"
cluster_addr = "https://first.example.com:8201"
plugin_directory = "/etc/vault.d/vault_plugins"

storage "raft" {
  path = "/opt/vault/data"
  node_id = "first"

  retry_join {
    leader_api_addr = "https://second.example.com:8200"
  }
  retry_join {
    leader_api_addr = "https://third.example.com:8200"
  }
}

# HTTPS listener
listener "tcp" {
  address       = "first.example.com:8200"
  cluster_addr = "first.example.com:8201"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
}

seal "transit" {
  address = "https://standalone.example.com:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
}

[verkaufer]

@hgarrow If you're using the Raft protocol, it sounds like this may be an issue with the Vault configuration.

Hashicorp has a section in their plugin documentation about the error message you're receiving:
https://www.vaultproject.io/docs/internals/plugins#unrecognized-remote-plugin-message

and they mention either disabling mlock or allowing the plugin to use mlock:

Disabling mlock when using Raft: https://www.vaultproject.io/docs/configuration#disable_mlock

Allowing the plugin to use mlock if you are using Raft and don't want to disable mlock:

sudo setcap cap_ipc_lock=+ep <plugin-binary>

Are you able to make those configuration changes? I'm suspecting this is the cause of the issue.

[hgarrow]

For the Raft implementation, mlock is disabled. I did use setcap against the binary on all three servers in the raft cluster and restarted vault, registering and creating a new secrets path, but still receive the same message when attempting to write the config to vault. From: David Gunter ***@***.***> Sent: Monday, August 2, 2021 12:22 PM To: 1Password/vault-plugin-secrets-onepassword ***@***.***> Cc: Garrow, Heather ***@***.***>; Mention ***@***.***> Subject: Re: [1Password/vault-plugin-secrets-onepassword] vault 1.5.0 integration errors (#9)

…

----------------------------------------------------------------------------------------------------------- CAUTION: This email originated from outside of the organization. DO NOT click links or open attachments unless you recognize the sender AND the sender's email address AND you know the content is safe. ----------------------------------------------------------------------------------------------------------- @hgarrow<https://github.com/hgarrow> If you're using the Raft protocol, it sounds like this may be an issue with the Vault configuration. Hashicorp has a section in their plugin documentation about the error message you're receiving: https://www.vaultproject.io/docs/internals/plugins#unrecognized-remote-plugin-message and they mention either disabling mlock or allowing the plugin to use mlock: Disabling mlock when using Raft: https://www.vaultproject.io/docs/configuration#disable_mlock Allowing the plugin to use mlock if you are using Raft and don't want to disable mlock: sudo setcap cap_ipc_lock=+ep <plugin-binary> Are you able to make those configuration changes? I'm suspecting this is the cause of the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#9 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AG7AOUBF6ABABVPGDICD253T23A3ZANCNFSM44SCII2A>.

[verkaufer]

Thanks for that information! Really appreciate your help getting to the bottom of this.

I noticed in the logs you posted earlier (#9 (comment) ) that Vault returned a 403: Permission denied when the plugin tried to initialize TLS with Vault.

Could you check the token you're using is allowed to unwrap responses?

Another angle to check is the api_addr / cluster_addr config values, but those look correct to me 🤔 (src)

I'm going to ask a few of my colleagues and we'll try to get you an answer as soon as possible.

[hgarrow]

I will admit, some of those earlier errors(#9) happened while we were attempting to get the connector docker containers working properly. The specific 403 error I am no longer seeing, though from the interactive debug session yesterday, we determined that in the Raft case, the traffic for the connection wasn't even reaching the docker environment when I was attempting to write the configuration (vault write op/config @/etc/vault.d/op-connect-config.json).

I make sure to use the same token and URI for the standalone vault as in the raft cluster, so I can confirm that the standalone vault is properly communicating, but the raft is not.

I did attempt to disable mlock on the standalone vault instance, but the 1password connection still worked afterwards.

[jillianwilson]

Hi again. David and I have been able to replicate your issue but have not been able to find the root of the problem. We have reached out to Hashicorp to see if they have any insight into what could be the issue and are awaiting a response.

[jillianwilson]

Hi again!

Just wanted to reach out and ask if you were ever able to resolve this issue? when we first looked at this last year we were unable to get a resolution from Hashicorp but we would like to look into this again if this is something you are still dealing with.